2024-02-10T05:56:34.516 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T05:56:34.516 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555113 --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_yuriw@teuthology --verbose --name yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555113/orig.config.yaml 2024-02-10T05:56:34.568 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T05:56:34.653 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555113 branch: main description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7555113' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 1610 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKNWJNri5/M3I44SbKPJiPkkMOmm4H0RVWDm51OdoTrgIXiuOjVCXG9a8WnXhNRl5emMgExYGPXc5QV2oedgbmM= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0pcWUwoBLylHUgQg13zjimT+TLcLEXzDbZlDAy+TtiwziBPT0LuA34roO+1Pb4MX74UMg/Fkq1knHb4ol8pmg= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/yuriw/src/github.com_ceph_ceph-c_main/qa/suites/orch/cephadm/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:44:16 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-10T05:56:34.653 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-10T05:56:34.654 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-10T05:56:34.654 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T05:56:34.656 INFO:teuthology.task.internal:Checking packages... 2024-02-10T05:56:34.680 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-10T05:56:34.681 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T05:56:34.681 INFO:teuthology.packaging:ref: None 2024-02-10T05:56:34.681 INFO:teuthology.packaging:tag: None 2024-02-10T05:56:34.681 INFO:teuthology.packaging:branch: main 2024-02-10T05:56:34.681 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:56:34.681 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-10T05:56:34.916 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-10T05:56:34.917 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T05:56:34.923 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T05:56:34.923 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T05:56:34.946 INFO:teuthology.task.internal:Saving configuration 2024-02-10T05:56:34.966 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T05:56:34.971 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:56:34.996 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555113', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:51:37.384077', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKNWJNri5/M3I44SbKPJiPkkMOmm4H0RVWDm51OdoTrgIXiuOjVCXG9a8WnXhNRl5emMgExYGPXc5QV2oedgbmM='} 2024-02-10T05:56:35.017 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555113', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:51:37.383183', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0pcWUwoBLylHUgQg13zjimT+TLcLEXzDbZlDAy+TtiwziBPT0LuA34roO+1Pb4MX74UMg/Fkq1knHb4ol8pmg='} 2024-02-10T05:56:35.017 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T05:56:35.023 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-10T05:56:35.023 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-10T05:56:35.023 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T05:56:35.094 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fad0d2f9550>, signals=[15]) 2024-02-10T05:56:35.095 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T05:56:35.100 INFO:teuthology.task.internal:Opening connections... 2024-02-10T05:56:35.100 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T05:56:35.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:56:35.193 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-02-10T05:56:35.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:56:35.282 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T05:56:35.288 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-02-10T05:56:35.305 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-02-10T05:56:35.306 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-02-10T05:56:35.362 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2024-02-10T05:56:35.362 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="9" 2024-02-10T05:56:35.362 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2024-02-10T05:56:35.362 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:56:35.362 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="9" 2024-02-10T05:56:35.362 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:56:35.362 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:56:35.362 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2024-02-10T05:56:35.362 INFO:teuthology.orchestra.run.smithi089.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:56:35.363 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:56:35.363 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:56:35.363 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:56:35.363 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:56:35.363 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:56:35.363 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-02-10T05:56:35.390 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-02-10T05:56:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-02-10T05:56:35.408 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-02-10T05:56:35.465 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-02-10T05:56:35.465 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-02-10T05:56:35.466 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-02-10T05:56:35.466 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:56:35.466 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-02-10T05:56:35.466 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:56:35.466 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:56:35.466 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-02-10T05:56:35.466 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:56:35.466 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:56:35.466 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:56:35.466 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:56:35.467 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:56:35.467 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:56:35.467 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-02-10T05:56:35.499 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T05:56:35.507 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T05:56:35.512 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T05:56:35.512 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:56:35.514 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:56:35.530 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T05:56:35.535 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T05:56:35.535 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:56:35.570 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:56:35.659 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T05:56:35.712 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T05:56:35.726 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T05:56:35.726 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T05:56:35.727 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:56:35.727 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:56:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:56:35.743 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-02-10T05:56:35.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:56:35.744 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-02-10T05:56:35.800 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:56:35.800 INFO:teuthology.task.kernel:Running kernel on smithi089: 5.14.0-361.el9.x86_64 2024-02-10T05:56:35.800 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum install -y kernel 2024-02-10T05:56:35.802 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:56:35.802 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-361.el9.x86_64 2024-02-10T05:56:35.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-02-10T05:56:38.092 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-02-10T05:56:38.109 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-02-10T05:56:39.717 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-02-10T05:56:39.743 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-02-10T05:56:44.641 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.0 MB 00:04 2024-02-10T05:56:45.983 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 1.4 MB/s | 8.0 MB 00:05 2024-02-10T05:56:47.733 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 19 MB 00:02 2024-02-10T05:56:48.000 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-02-10T05:56:52.057 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 5.9 MB 00:00 2024-02-10T05:56:52.575 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 5.9 MB 00:01 2024-02-10T05:56:53.611 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-02-10T05:56:54.117 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-02-10T05:56:56.422 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 20 MB 00:02 2024-02-10T05:57:01.245 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-02-10T05:57:03.543 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:57:03.543 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:57:03.642 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T05:57:03.647 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T05:57:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout:Install 4 Packages 2024-02-10T05:57:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T05:57:03.649 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 99 M 2024-02-10T05:57:03.649 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 122 M 2024-02-10T05:57:03.649 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T05:57:04.038 INFO:teuthology.orchestra.run.smithi089.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-02-10T05:57:04.696 INFO:teuthology.orchestra.run.smithi089.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 20 MB/s | 21 MB 00:01 2024-02-10T05:57:05.246 INFO:teuthology.orchestra.run.smithi089.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 28 MB/s | 33 MB 00:01 2024-02-10T05:57:05.755 INFO:teuthology.orchestra.run.smithi089.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 19 MB/s | 39 MB 00:02 2024-02-10T05:57:05.758 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:57:05.758 INFO:teuthology.orchestra.run.smithi089.stdout:Total 47 MB/s | 99 MB 00:02 2024-02-10T05:57:05.759 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T05:57:06.151 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T05:57:06.152 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T05:57:07.223 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T05:57:07.223 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T05:57:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T05:57:10.262 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:57:10.380 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:57:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:57:13.309 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:57:13.458 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 20 MB 00:19 2024-02-10T05:57:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:57:18.255 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:57:18.408 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-02-10T05:57:20.681 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:57:20.682 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:57:20.765 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T05:57:20.774 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T05:57:20.774 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-10T05:57:20.774 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T05:57:20.774 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T05:57:20.774 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:57:20.775 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-10T05:57:20.775 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:57:20.775 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:57:20.775 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:57:20.775 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T05:57:20.775 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T05:57:20.775 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T05:57:20.775 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-02-10T05:57:20.775 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T05:57:20.775 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 99 M 2024-02-10T05:57:20.775 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 122 M 2024-02-10T05:57:20.776 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T05:57:21.259 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 12 MB/s | 6.0 MB 00:00 2024-02-10T05:57:21.634 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-02-10T05:57:22.268 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 26 MB/s | 39 MB 00:01 2024-02-10T05:57:22.752 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 22 MB/s | 33 MB 00:01 2024-02-10T05:57:22.755 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:57:22.755 INFO:teuthology.orchestra.run.smithi173.stdout:Total 50 MB/s | 99 MB 00:01 2024-02-10T05:57:22.755 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T05:57:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:57:23.128 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T05:57:23.128 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T05:57:24.142 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T05:57:24.143 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T05:57:26.546 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T05:57:27.252 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:57:27.385 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:57:30.189 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:57:30.379 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:57:35.075 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:57:35.291 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:57:40.045 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi089.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T05:57:52.586 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:57:53.038 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:57:53.038 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:57:53.038 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:57:53.038 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:57:53.451 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:57:53.451 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T05:57:53.452 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T05:57:53.452 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:57:53.452 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:57:53.452 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:57:53.452 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:57:53.452 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T05:57:53.452 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T05:57:53.969 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:57:54.119 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:57:54.119 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:57:54.120 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:57:54.120 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:57:54.120 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:57:54.120 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-10T05:57:54.120 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi089.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:57:54.120 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:57:54.227 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:57:54.227 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:57:54.227 DEBUG:teuthology.orchestra.run.smithi089:> sudo rpm -qi grub2-tools 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:Name : grub2-tools 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:Epoch : 1 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:Version : 2.06 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:Release : 68.el9 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:Architecture: x86_64 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:Group : Unspecified 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:Size : 8264254 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:License : GPLv3+ 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:57:54.269 INFO:teuthology.orchestra.run.smithi089.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout:Packager : builder@centos.org 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout:Vendor : CentOS 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout:Summary : Support tools for GRUB. 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout:Description : 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout:hardware devices. 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T05:57:54.270 INFO:teuthology.orchestra.run.smithi089.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:57:54.271 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:57:54.271 INFO:teuthology.task.kernel:Updating grub on smithi089 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:57:54.272 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:57:54.888 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2024-02-10T05:57:56.299 INFO:teuthology.orchestra.run.smithi089.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:57:56.332 INFO:teuthology.orchestra.run.smithi089.stderr:done 2024-02-10T05:57:56.339 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-02-10T05:57:56.354 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.S2aBxkhKo2 2024-02-10T05:57:56.354 DEBUG:teuthology.orchestra.run.smithi089:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.S2aBxkhKo2 2024-02-10T05:57:56.423 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.S2aBxkhKo2 2024-02-10T05:57:56.559 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.S2aBxkhKo2 is 6KB 2024-02-10T05:57:56.607 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.S2aBxkhKo2 2024-02-10T05:57:56.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:57:56.690 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:57:56.691 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:57:56.691 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:57:56.691 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:57:56.692 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:57:56.849 DEBUG:teuthology.orchestra.run.smithi089:> sudo shutdown -r now 2024-02-10T05:58:09.699 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:58:09.700 INFO:teuthology.orchestra.run.smithi173.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:58:09.700 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T05:58:09.778 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:58:10.193 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:58:10.193 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:58:10.193 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:58:10.193 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:58:10.585 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:58:10.586 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T05:58:10.586 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T05:58:10.586 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:58:10.586 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:58:10.586 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:58:10.586 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:58:10.587 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T05:58:10.587 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T05:58:11.099 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:58:11.277 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:58:11.277 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:58:11.277 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:58:11.277 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:58:11.277 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:58:11.277 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-10T05:58:11.277 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:58:11.278 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:58:11.384 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:58:11.385 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:58:11.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-02-10T05:58:11.422 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-02-10T05:58:11.422 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-02-10T05:58:11.422 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-02-10T05:58:11.422 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 68.el9 2024-02-10T05:58:11.422 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-02-10T05:58:11.422 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:58:11.422 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-02-10T05:58:11.422 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8264254 2024-02-10T05:58:11.422 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:58:11.423 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:58:11.424 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-02-10T05:58:11.424 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T05:58:11.424 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:58:11.424 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:58:11.425 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:58:11.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:58:12.053 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-02-10T05:58:13.388 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:58:13.423 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-02-10T05:58:13.425 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-02-10T05:58:13.440 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.yk9qdz1EOv 2024-02-10T05:58:13.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yk9qdz1EOv 2024-02-10T05:58:13.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.yk9qdz1EOv 2024-02-10T05:58:13.661 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.yk9qdz1EOv is 6KB 2024-02-10T05:58:13.710 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.yk9qdz1EOv 2024-02-10T05:58:13.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:58:13.790 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:58:13.790 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:58:13.790 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:58:13.790 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:58:13.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:58:13.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-02-10T05:58:26.856 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-10T05:58:26.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-02-10T05:58:26.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:58:43.976 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-10T05:58:43.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T05:58:43.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:59:26.902 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T05:59:35.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-02-10T05:59:35.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:59:37.408 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-02-10T05:59:37.875 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-02-10T05:59:37.875 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:59:37.875 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-02-10T05:59:37.889 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:59:37.889 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:59:37.889 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:59:37.890 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:59:38.890 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-10T05:59:38.890 DEBUG:teuthology.orchestra.run.smithi089:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:59:38.991 INFO:teuthology.orchestra.run.smithi089.stdout:ttyS1 2024-02-10T05:59:39.023 DEBUG:teuthology.parallel:result is None 2024-02-10T05:59:43.979 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T05:59:52.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T05:59:52.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:59:56.456 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-02-10T05:59:56.936 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T05:59:56.936 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:59:56.937 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-02-10T05:59:56.950 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:59:56.951 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:59:56.951 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:59:56.951 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:59:57.951 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-10T05:59:57.951 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:59:58.025 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-02-10T05:59:58.055 DEBUG:teuthology.parallel:result is None 2024-02-10T05:59:58.055 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T05:59:58.062 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T05:59:58.062 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:59:58.066 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:59:58.082 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T05:59:58.090 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T05:59:58.099 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T05:59:58.099 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:59:58.124 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:59:58.159 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T05:59:58.167 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T05:59:58.168 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:59:58.187 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:59:58.218 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:59:58.232 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:59:58.238 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:59:58.251 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:59:58.252 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T05:59:58.259 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T05:59:58.260 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:59:58.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:59:58.320 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T05:59:58.330 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T05:59:58.330 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:59:58.345 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:59:58.374 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:59:58.444 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:59:58.491 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:59:58.569 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:59:58.615 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T05:59:58.615 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:59:58.682 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:59:58.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:59:58.773 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:59:58.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:59:58.897 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T05:59:58.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:59:58.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-02-10T05:59:58.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-02-10T05:59:59.025 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:59:59.050 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:59:59.389 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T05:59:59.400 INFO:teuthology.task.internal:Starting timer... 2024-02-10T05:59:59.401 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T05:59:59.418 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T05:59:59.431 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-10T05:59:59.431 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2024-02-10T05:59:59.482 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2024-02-10T05:59:59.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-02-10T05:59:59.541 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-02-10T05:59:59.543 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T05:59:59.543 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2024-02-10T05:59:59.580 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2024-02-10T05:59:59.580 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-02-10T05:59:59.598 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-02-10T05:59:59.598 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi089.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-02-10T05:59:59.599 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:59:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:59:59.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:59:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:59:59.675 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T05:59:59.675 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:59:59.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:59:59.758 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T05:59:59.771 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T05:59:59.794 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T05:59:59.795 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi089.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T06:04:11.034 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi089.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-02-10T06:04:11.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-02-10T06:04:11.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:04:11.129 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-02-10T06:04:11.207 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-02-10T06:04:11.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:04:11.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:04:11.299 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-02-10T06:04:11.383 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:04:11.384 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T06:04:11.395 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T06:04:11.395 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T06:04:11.396 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:04:11.398 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T06:04:11.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:04:11.428 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T06:04:11.445 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T06:04:11.454 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T06:04:11.471 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T06:04:11.474 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2024-02-10T06:04:11.487 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2024-02-10T06:04:11.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-02-10T06:04:11.505 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T06:04:11.514 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-02-10T06:04:11.521 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T06:04:11.531 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T06:04:11.547 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T06:04:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:04:11.578 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:04:11.578 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-02-10T06:04:11.578 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:04:11.578 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:04:11.578 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:04:11.578 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:04:11.597 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:04:11.600 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:04:11.600 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-02-10T06:04:11.600 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:04:11.600 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:04:11.600 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:04:11.601 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:04:11.601 INFO:teuthology.run_tasks:Running task install... 2024-02-10T06:04:11.610 DEBUG:teuthology.task.install:project ceph 2024-02-10T06:04:11.610 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}} 2024-02-10T06:04:11.610 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T06:04:11.611 INFO:teuthology.task.install:Using flavor: default 2024-02-10T06:04:11.617 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-10T06:04:11.617 INFO:teuthology.task.install:extra packages: [] 2024-02-10T06:04:11.617 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-10T06:04:11.617 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:04:11.617 INFO:teuthology.packaging:ref: None 2024-02-10T06:04:11.618 INFO:teuthology.packaging:tag: None 2024-02-10T06:04:11.618 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:04:11.618 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:04:11.618 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-10T06:04:11.620 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-10T06:04:11.620 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:04:11.620 INFO:teuthology.packaging:ref: None 2024-02-10T06:04:11.620 INFO:teuthology.packaging:tag: None 2024-02-10T06:04:11.620 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:04:11.620 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:04:11.620 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-10T06:04:11.857 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/ 2024-02-10T06:04:11.857 INFO:teuthology.task.install.rpm:Package version is 17.2.7-505.g72ce7c1d 2024-02-10T06:04:11.873 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/ 2024-02-10T06:04:11.873 INFO:teuthology.task.install.rpm:Package version is 17.2.7-505.g72ce7c1d 2024-02-10T06:04:11.995 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T06:04:11.996 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:04:11.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T06:04:12.004 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T06:04:12.004 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:04:12.004 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T06:04:12.024 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T06:04:12.024 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:04:12.024 INFO:teuthology.packaging:ref: None 2024-02-10T06:04:12.024 INFO:teuthology.packaging:tag: None 2024-02-10T06:04:12.024 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:04:12.024 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:04:12.025 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T06:04:12.033 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T06:04:12.033 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:04:12.033 INFO:teuthology.packaging:ref: None 2024-02-10T06:04:12.033 INFO:teuthology.packaging:tag: None 2024-02-10T06:04:12.034 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:04:12.034 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:04:12.034 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T06:04:12.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T06:04:12.101 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T06:04:12.173 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T06:04:12.181 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T06:04:12.239 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-02-10T06:04:12.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-02-10T06:04:12.247 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2024-02-10T06:04:12.249 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2024-02-10T06:04:12.477 INFO:teuthology.orchestra.run.smithi173.stdout:74 files removed 2024-02-10T06:04:12.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-02-10T06:04:12.511 INFO:teuthology.orchestra.run.smithi089.stdout:74 files removed 2024-02-10T06:04:12.538 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw 2024-02-10T06:04:13.179 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 218 kB/s | 77 kB 00:00 2024-02-10T06:04:13.194 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 229 kB/s | 77 kB 00:00 2024-02-10T06:04:13.554 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 28 kB/s | 9.1 kB 00:00 2024-02-10T06:04:13.560 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 29 kB/s | 9.1 kB 00:00 2024-02-10T06:04:13.911 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 5.7 kB/s | 1.8 kB 00:00 2024-02-10T06:04:13.913 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 5.6 kB/s | 1.8 kB 00:00 2024-02-10T06:04:14.699 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 9.0 MB/s | 6.7 MB 00:00 2024-02-10T06:04:14.813 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - AppStream 7.8 MB/s | 6.7 MB 00:00 2024-02-10T06:04:16.275 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-02-10T06:04:16.280 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-02-10T06:04:17.071 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-02-10T06:04:18.465 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.0 MB 00:01 2024-02-10T06:04:20.700 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-02-10T06:04:21.347 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 19 MB 00:03 2024-02-10T06:04:26.390 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 5.9 MB 00:01 2024-02-10T06:04:26.884 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 5.9 MB 00:02 2024-02-10T06:04:27.840 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 15 kB 00:00 2024-02-10T06:04:28.201 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 92 kB/s | 15 kB 00:00 2024-02-10T06:04:29.111 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 26 MB/s | 20 MB 00:00 2024-02-10T06:04:32.027 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 5.0 MB/s | 20 MB 00:04 2024-02-10T06:04:34.036 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-02-10T06:04:36.654 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:04:36.665 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:04:36.665 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-10T06:04:36.665 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:04:36.665 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:04:36.666 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 14 M 2024-02-10T06:04:36.666 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-02-10T06:04:36.666 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.1 M 2024-02-10T06:04:36.666 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-10T06:04:36.666 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-10T06:04:36.666 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 5.0 M 2024-02-10T06:04:36.666 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 23 M 2024-02-10T06:04:36.666 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-10T06:04:36.666 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-10T06:04:36.666 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 635 k 2024-02-10T06:04:36.666 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 458 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 4.9 M 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 45 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 116 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 156 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 320 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 293 k 2024-02-10T06:04:36.667 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 99 k 2024-02-10T06:04:36.668 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-10T06:04:36.668 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:04:36.668 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:04:36.668 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:04:36.668 INFO:teuthology.orchestra.run.smithi173.stdout:Install 22 Packages 2024-02-10T06:04:36.668 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-02-10T06:04:36.668 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:04:36.668 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 58 M 2024-02-10T06:04:36.669 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:04:36.850 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-02-10T06:04:37.804 INFO:teuthology.orchestra.run.smithi173.stdout:(1/24): ceph-base-17.2.7-505.g72ce7c1d.el9.x86_ 6.4 MB/s | 5.0 MB 00:00 2024-02-10T06:04:37.879 INFO:teuthology.orchestra.run.smithi173.stdout:(2/24): ceph-selinux-17.2.7-505.g72ce7c1d.el9.x 335 kB/s | 25 kB 00:00 2024-02-10T06:04:37.979 INFO:teuthology.orchestra.run.smithi173.stdout:(3/24): libcephfs2-17.2.7-505.g72ce7c1d.el9.x86 6.2 MB/s | 635 kB 00:00 2024-02-10T06:04:38.088 INFO:teuthology.orchestra.run.smithi173.stdout:(4/24): libradosstriper1-17.2.7-505.g72ce7c1d.e 4.1 MB/s | 458 kB 00:00 2024-02-10T06:04:38.806 INFO:teuthology.orchestra.run.smithi173.stdout:(5/24): librgw2-17.2.7-505.g72ce7c1d.el9.x86_64 6.8 MB/s | 4.9 MB 00:00 2024-02-10T06:04:38.948 INFO:teuthology.orchestra.run.smithi173.stdout:(6/24): ceph-radosgw-17.2.7-505.g72ce7c1d.el9.x 7.3 MB/s | 14 MB 00:01 2024-02-10T06:04:38.973 INFO:teuthology.orchestra.run.smithi173.stdout:(7/24): python3-ceph-argparse-17.2.7-505.g72ce7 270 kB/s | 45 kB 00:00 2024-02-10T06:04:38.998 INFO:teuthology.orchestra.run.smithi173.stdout:(8/24): python3-ceph-common-17.2.7-505.g72ce7c1 2.3 MB/s | 116 kB 00:00 2024-02-10T06:04:39.032 INFO:teuthology.orchestra.run.smithi173.stdout:(9/24): python3-cephfs-17.2.7-505.g72ce7c1d.el9 2.6 MB/s | 156 kB 00:00 2024-02-10T06:04:39.265 INFO:teuthology.orchestra.run.smithi173.stdout:(10/24): ceph-common-17.2.7-505.g72ce7c1d.el9.x 10 MB/s | 23 MB 00:02 2024-02-10T06:04:39.308 INFO:teuthology.orchestra.run.smithi173.stdout:(11/24): python3-rados-17.2.7-505.g72ce7c1d.el9 1.0 MB/s | 320 kB 00:00 2024-02-10T06:04:39.341 INFO:teuthology.orchestra.run.smithi173.stdout:(12/24): python3-rbd-17.2.7-505.g72ce7c1d.el9.x 948 kB/s | 293 kB 00:00 2024-02-10T06:04:39.366 INFO:teuthology.orchestra.run.smithi173.stdout:(13/24): python3-rgw-17.2.7-505.g72ce7c1d.el9.x 993 kB/s | 99 kB 00:00 2024-02-10T06:04:39.500 INFO:teuthology.orchestra.run.smithi173.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 411 kB/s | 44 kB 00:00 2024-02-10T06:04:39.510 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:04:39.525 INFO:teuthology.orchestra.run.smithi173.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 733 kB/s | 159 kB 00:00 2024-02-10T06:04:39.528 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:04:39.528 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-02-10T06:04:39.528 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:04:39.528 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:04:39.528 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 14 M 2024-02-10T06:04:39.528 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-02-10T06:04:39.528 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.1 M 2024-02-10T06:04:39.528 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 5.0 M 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 23 M 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 635 k 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 458 k 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 4.9 M 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-10T06:04:39.529 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 45 k 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 116 k 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 156 k 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 320 k 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 293 k 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 99 k 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout:Install 22 Packages 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 2 Packages 2024-02-10T06:04:39.530 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:04:39.531 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 58 M 2024-02-10T06:04:39.531 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:04:39.558 INFO:teuthology.orchestra.run.smithi173.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-02-10T06:04:39.575 INFO:teuthology.orchestra.run.smithi173.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2024-02-10T06:04:39.601 INFO:teuthology.orchestra.run.smithi173.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 661 kB 00:00 2024-02-10T06:04:40.434 INFO:teuthology.orchestra.run.smithi089.stdout:(1/24): ceph-base-17.2.7-505.g72ce7c1d.el9.x86_ 6.1 MB/s | 5.0 MB 00:00 2024-02-10T06:04:40.493 INFO:teuthology.orchestra.run.smithi089.stdout:(2/24): ceph-selinux-17.2.7-505.g72ce7c1d.el9.x 432 kB/s | 25 kB 00:00 2024-02-10T06:04:40.626 INFO:teuthology.orchestra.run.smithi089.stdout:(3/24): libcephfs2-17.2.7-505.g72ce7c1d.el9.x86 4.7 MB/s | 635 kB 00:00 2024-02-10T06:04:40.693 INFO:teuthology.orchestra.run.smithi173.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 43 kB/s | 49 kB 00:01 2024-02-10T06:04:40.727 INFO:teuthology.orchestra.run.smithi089.stdout:(4/24): libradosstriper1-17.2.7-505.g72ce7c1d.e 4.5 MB/s | 458 kB 00:00 2024-02-10T06:04:40.785 INFO:teuthology.orchestra.run.smithi173.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 57 kB/s | 67 kB 00:01 2024-02-10T06:04:41.168 INFO:teuthology.orchestra.run.smithi173.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 192 kB/s | 309 kB 00:01 2024-02-10T06:04:41.510 INFO:teuthology.orchestra.run.smithi089.stdout:(5/24): librgw2-17.2.7-505.g72ce7c1d.el9.x86_64 6.3 MB/s | 4.9 MB 00:00 2024-02-10T06:04:41.594 INFO:teuthology.orchestra.run.smithi089.stdout:(6/24): python3-ceph-argparse-17.2.7-505.g72ce7 542 kB/s | 45 kB 00:00 2024-02-10T06:04:41.627 INFO:teuthology.orchestra.run.smithi173.stdout:(22/24): librados2-17.2.7-505.g72ce7c1d.el9.x86 3.7 MB/s | 3.1 MB 00:00 2024-02-10T06:04:41.727 INFO:teuthology.orchestra.run.smithi173.stdout:(23/24): thrift-0.15.0-2.el9.x86_64.rpm 1.5 MB/s | 1.6 MB 00:01 2024-02-10T06:04:41.728 INFO:teuthology.orchestra.run.smithi089.stdout:(7/24): ceph-radosgw-17.2.7-505.g72ce7c1d.el9.x 6.6 MB/s | 14 MB 00:02 2024-02-10T06:04:41.752 INFO:teuthology.orchestra.run.smithi089.stdout:(8/24): python3-ceph-common-17.2.7-505.g72ce7c1 733 kB/s | 116 kB 00:00 2024-02-10T06:04:41.786 INFO:teuthology.orchestra.run.smithi089.stdout:(9/24): python3-cephfs-17.2.7-505.g72ce7c1d.el9 2.6 MB/s | 156 kB 00:00 2024-02-10T06:04:41.820 INFO:teuthology.orchestra.run.smithi089.stdout:(10/24): python3-rados-17.2.7-505.g72ce7c1d.el9 4.7 MB/s | 320 kB 00:00 2024-02-10T06:04:41.853 INFO:teuthology.orchestra.run.smithi089.stdout:(11/24): python3-rbd-17.2.7-505.g72ce7c1d.el9.x 4.3 MB/s | 293 kB 00:00 2024-02-10T06:04:41.879 INFO:teuthology.orchestra.run.smithi089.stdout:(12/24): python3-rgw-17.2.7-505.g72ce7c1d.el9.x 1.7 MB/s | 99 kB 00:00 2024-02-10T06:04:42.113 INFO:teuthology.orchestra.run.smithi089.stdout:(13/24): ceph-common-17.2.7-505.g72ce7c1d.el9.x 9.0 MB/s | 23 MB 00:02 2024-02-10T06:04:42.188 INFO:teuthology.orchestra.run.smithi089.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 476 kB/s | 159 kB 00:00 2024-02-10T06:04:42.205 INFO:teuthology.orchestra.run.smithi089.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 146 kB/s | 44 kB 00:00 2024-02-10T06:04:42.230 INFO:teuthology.orchestra.run.smithi089.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-02-10T06:04:42.255 INFO:teuthology.orchestra.run.smithi089.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-02-10T06:04:42.272 INFO:teuthology.orchestra.run.smithi089.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 621 kB/s | 41 kB 00:00 2024-02-10T06:04:42.447 INFO:teuthology.orchestra.run.smithi089.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-02-10T06:04:42.472 INFO:teuthology.orchestra.run.smithi089.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 336 kB/s | 67 kB 00:00 2024-02-10T06:04:42.514 INFO:teuthology.orchestra.run.smithi089.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-02-10T06:04:42.765 INFO:teuthology.orchestra.run.smithi089.stdout:(22/24): librbd1-17.2.7-505.g72ce7c1d.el9.x86_6 12 MB/s | 2.9 MB 00:00 2024-02-10T06:04:42.823 INFO:teuthology.orchestra.run.smithi089.stdout:(23/24): librados2-17.2.7-505.g72ce7c1d.el9.x86 8.8 MB/s | 3.1 MB 00:00 2024-02-10T06:04:43.023 INFO:teuthology.orchestra.run.smithi089.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 2.7 MB/s | 1.6 MB 00:00 2024-02-10T06:04:43.029 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:04:43.029 INFO:teuthology.orchestra.run.smithi089.stdout:Total 17 MB/s | 58 MB 00:03 2024-02-10T06:04:43.029 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:04:43.057 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:04:43.057 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:04:43.516 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:04:43.516 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:04:43.794 INFO:teuthology.orchestra.run.smithi173.stdout:(24/24): librbd1-17.2.7-505.g72ce7c1d.el9.x86_6 1.1 MB/s | 2.9 MB 00:02 2024-02-10T06:04:43.804 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:04:43.804 INFO:teuthology.orchestra.run.smithi173.stdout:Total 8.1 MB/s | 58 MB 00:07 2024-02-10T06:04:43.804 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:04:43.836 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:04:43.836 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:04:44.296 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:04:44.296 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:04:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:04:45.063 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-10T06:04:45.126 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:04:45.209 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:04:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-10T06:04:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:04:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:04:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-10T06:04:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-10T06:04:45.518 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:04:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-10T06:04:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:04:45.897 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-10T06:04:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-10T06:04:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-10T06:04:45.979 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:04:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-10T06:04:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 10/26 2024-02-10T06:04:46.060 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-10T06:04:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-10T06:04:46.089 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:04:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-10T06:04:46.158 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:04:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:04:46.206 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:04:46.218 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-10T06:04:46.246 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-10T06:04:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-10T06:04:46.282 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-10T06:04:46.315 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-10T06:04:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-10T06:04:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-10T06:04:46.688 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-10T06:04:46.727 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-10T06:04:46.747 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-10T06:04:46.799 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 19/26 2024-02-10T06:04:46.813 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-10T06:04:46.822 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 20/26 2024-02-10T06:04:46.842 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-10T06:04:46.884 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 10/26 2024-02-10T06:04:46.922 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-10T06:04:46.983 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-10T06:04:47.012 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:04:47.077 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:04:47.117 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-10T06:04:47.153 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-10T06:04:47.186 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-10T06:04:47.482 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-10T06:04:47.516 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-10T06:04:47.605 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-10T06:04:47.678 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 19/26 2024-02-10T06:04:47.702 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 20/26 2024-02-10T06:04:49.328 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:04:49.369 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:04:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:04:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-10T06:04:50.036 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-10T06:04:50.036 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-10T06:04:50.036 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-10T06:04:50.036 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:04:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:04:50.123 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:04:50.166 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:04:50.708 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-10T06:04:50.745 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-10T06:04:50.840 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-10T06:04:50.840 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-10T06:04:50.841 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-10T06:04:50.841 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:04:50.871 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:04:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:04:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2024-02-10T06:04:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2024-02-10T06:04:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2024-02-10T06:04:59.619 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2024-02-10T06:04:59.619 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2024-02-10T06:04:59.619 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /root 2024-02-10T06:04:59.619 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2024-02-10T06:04:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:00.389 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:05:00.389 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-02-10T06:05:00.389 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-02-10T06:05:00.389 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-02-10T06:05:00.389 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-02-10T06:05:00.389 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-02-10T06:05:00.389 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-02-10T06:05:00.390 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-02-10T06:05:00.390 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:00.897 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-10T06:05:00.974 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-10T06:05:00.974 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:00.974 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-10T06:05:00.974 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T06:05:00.974 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T06:05:00.975 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:00.976 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-10T06:05:01.066 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-10T06:05:01.067 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:05:01.710 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-10T06:05:01.785 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-10T06:05:01.785 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:01.786 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-10T06:05:01.786 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T06:05:01.786 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T06:05:01.786 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:01.787 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-10T06:05:01.877 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-10T06:05:01.877 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:05:02.339 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 7/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 8/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 9/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 10/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-10T06:05:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-10T06:05:02.341 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 25/26 2024-02-10T06:05:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:05:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/26 2024-02-10T06:05:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-10T06:05:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-10T06:05:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-10T06:05:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/26 2024-02-10T06:05:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/26 2024-02-10T06:05:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 7/26 2024-02-10T06:05:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 8/26 2024-02-10T06:05:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 9/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 10/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-10T06:05:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-10T06:05:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-10T06:05:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 25/26 2024-02-10T06:05:03.264 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:05:03.264 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:03.264 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-02-10T06:05:03.264 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.264 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.264 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-10T06:05:03.265 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-10T06:05:03.266 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.266 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.266 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.266 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-10T06:05:03.266 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.266 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.266 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:03.266 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-10T06:05:03.266 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:03.266 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:05:03.635 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-test 2024-02-10T06:05:04.027 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-10T06:05:04.027 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:04.027 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-02-10T06:05:04.027 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.028 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:04.029 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:05:04.209 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:28 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:05:04.280 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-02-10T06:05:04.370 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:05:04.370 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:04.370 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 48 M 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 162 k 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-02-10T06:05:04.371 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:04.372 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 49 M 2024-02-10T06:05:04.372 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 196 M 2024-02-10T06:05:04.372 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:05:04.536 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 300 kB 00:00 2024-02-10T06:05:04.570 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-10T06:05:04.612 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.3 MB/s | 186 kB 00:00 2024-02-10T06:05:04.654 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-02-10T06:05:04.729 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): libcephsqlite-17.2.7-505.g72ce7c1d.el9.x 455 kB/s | 162 kB 00:00 2024-02-10T06:05:04.859 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:30 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:05:05.017 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 48 M 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 162 k 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:05:05.018 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:05.019 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-02-10T06:05:05.019 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:05.019 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 49 M 2024-02-10T06:05:05.019 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 196 M 2024-02-10T06:05:05.019 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:05:05.190 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-02-10T06:05:05.224 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-10T06:05:05.266 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-10T06:05:05.300 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.4 MB/s | 217 kB 00:00 2024-02-10T06:05:05.350 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): libcephsqlite-17.2.7-505.g72ce7c1d.el9.x 492 kB/s | 162 kB 00:00 2024-02-10T06:05:07.988 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): ceph-test-17.2.7-505.g72ce7c1d.el9.x86_6 13 MB/s | 48 MB 00:03 2024-02-10T06:05:07.991 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:07.991 INFO:teuthology.orchestra.run.smithi089.stdout:Total 14 MB/s | 49 MB 00:03 2024-02-10T06:05:07.992 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:05:08.010 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:05:08.010 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:05:08.406 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:05:08.406 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:05:08.998 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:05:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-10T06:05:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-10T06:05:09.148 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-10T06:05:09.197 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-10T06:05:09.226 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-10T06:05:10.325 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): ceph-test-17.2.7-505.g72ce7c1d.el9.x86_6 9.0 MB/s | 48 MB 00:05 2024-02-10T06:05:10.327 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:10.327 INFO:teuthology.orchestra.run.smithi173.stdout:Total 9.2 MB/s | 49 MB 00:05 2024-02-10T06:05:10.328 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:05:10.343 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:05:10.343 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:05:10.728 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:05:10.728 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:05:11.326 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:05:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-10T06:05:11.433 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-10T06:05:11.476 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-10T06:05:11.525 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-10T06:05:11.553 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-10T06:05:13.673 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-10T06:05:13.750 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-10T06:05:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-10T06:05:16.033 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-10T06:05:16.104 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-10T06:05:16.104 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/6 2024-02-10T06:05:16.104 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/6 2024-02-10T06:05:16.104 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-10T06:05:16.105 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-10T06:05:16.105 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-10T06:05:16.527 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-10T06:05:16.527 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:16.527 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:05:16.527 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:16.527 INFO:teuthology.orchestra.run.smithi089.stdout: jq-1.6-16.el9.x86_64 2024-02-10T06:05:16.527 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:16.527 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-10T06:05:16.527 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-10T06:05:16.528 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-10T06:05:16.528 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:16.528 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:05:16.828 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph 2024-02-10T06:05:17.432 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:41 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:05:17.605 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:05:17.607 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-02-10T06:05:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 6.5 k 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.1 M 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 1.4 M 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 239 k 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.7 M 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 15 M 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-10T06:05:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-10T06:05:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:05:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-02-10T06:05:17.611 INFO:teuthology.orchestra.run.smithi089.stdout:Install 33 Packages 2024-02-10T06:05:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:17.611 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 27 M 2024-02-10T06:05:17.611 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 99 M 2024-02-10T06:05:17.611 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:05:17.858 INFO:teuthology.orchestra.run.smithi089.stdout:(1/33): ceph-17.2.7-505.g72ce7c1d.el9.x86_64.rp 38 kB/s | 6.5 kB 00:00 2024-02-10T06:05:18.158 INFO:teuthology.orchestra.run.smithi089.stdout:(2/33): ceph-mgr-17.2.7-505.g72ce7c1d.el9.x86_6 3.0 MB/s | 1.4 MB 00:00 2024-02-10T06:05:18.200 INFO:teuthology.orchestra.run.smithi089.stdout:(3/33): ceph-mds-17.2.7-505.g72ce7c1d.el9.x86_6 4.0 MB/s | 2.1 MB 00:00 2024-02-10T06:05:18.258 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-10T06:05:18.259 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/6 2024-02-10T06:05:18.259 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/6 2024-02-10T06:05:18.259 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-10T06:05:18.259 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-10T06:05:18.259 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-10T06:05:18.275 INFO:teuthology.orchestra.run.smithi089.stdout:(4/33): ceph-mgr-modules-core-17.2.7-505.g72ce7 3.1 MB/s | 239 kB 00:00 2024-02-10T06:05:18.384 INFO:teuthology.orchestra.run.smithi089.stdout:(5/33): ceph-mon-17.2.7-505.g72ce7c1d.el9.x86_6 7.0 MB/s | 3.7 MB 00:00 2024-02-10T06:05:18.426 INFO:teuthology.orchestra.run.smithi089.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 782 kB/s | 32 kB 00:00 2024-02-10T06:05:18.451 INFO:teuthology.orchestra.run.smithi089.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 822 kB/s | 144 kB 00:00 2024-02-10T06:05:18.468 INFO:teuthology.orchestra.run.smithi089.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-02-10T06:05:18.485 INFO:teuthology.orchestra.run.smithi089.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-02-10T06:05:18.510 INFO:teuthology.orchestra.run.smithi089.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-02-10T06:05:18.560 INFO:teuthology.orchestra.run.smithi089.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-02-10T06:05:18.585 INFO:teuthology.orchestra.run.smithi089.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-02-10T06:05:18.611 INFO:teuthology.orchestra.run.smithi089.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-10T06:05:18.637 INFO:teuthology.orchestra.run.smithi089.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-10T06:05:18.662 INFO:teuthology.orchestra.run.smithi089.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-02-10T06:05:18.696 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-10T06:05:18.696 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:18.697 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:05:18.697 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:18.697 INFO:teuthology.orchestra.run.smithi173.stdout: jq-1.6-16.el9.x86_64 2024-02-10T06:05:18.697 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:18.697 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-10T06:05:18.697 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-10T06:05:18.697 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-10T06:05:18.697 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:18.697 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:05:18.753 INFO:teuthology.orchestra.run.smithi089.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 372 kB/s | 43 kB 00:00 2024-02-10T06:05:18.812 INFO:teuthology.orchestra.run.smithi089.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-02-10T06:05:18.846 INFO:teuthology.orchestra.run.smithi089.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 319 kB/s | 11 kB 00:00 2024-02-10T06:05:18.888 INFO:teuthology.orchestra.run.smithi089.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-02-10T06:05:18.904 INFO:teuthology.orchestra.run.smithi089.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-02-10T06:05:18.921 INFO:teuthology.orchestra.run.smithi089.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 694 kB/s | 23 kB 00:00 2024-02-10T06:05:18.938 INFO:teuthology.orchestra.run.smithi089.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 577 kB/s | 19 kB 00:00 2024-02-10T06:05:18.955 INFO:teuthology.orchestra.run.smithi089.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 584 kB/s | 20 kB 00:00 2024-02-10T06:05:18.999 INFO:teuthology.orchestra.run.smithi089.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 761 kB/s | 46 kB 00:00 2024-02-10T06:05:19.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-02-10T06:05:19.025 INFO:teuthology.orchestra.run.smithi089.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-02-10T06:05:19.209 INFO:teuthology.orchestra.run.smithi089.stdout:(26/33): ceph-osd-17.2.7-505.g72ce7c1d.el9.x86_ 15 MB/s | 15 MB 00:01 2024-02-10T06:05:19.234 INFO:teuthology.orchestra.run.smithi089.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 79 kB/s | 16 kB 00:00 2024-02-10T06:05:19.259 INFO:teuthology.orchestra.run.smithi089.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.0 MB/s | 272 kB 00:00 2024-02-10T06:05:19.285 INFO:teuthology.orchestra.run.smithi089.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-02-10T06:05:19.343 INFO:teuthology.orchestra.run.smithi089.stdout:(30/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2024-02-10T06:05:19.368 INFO:teuthology.orchestra.run.smithi089.stdout:(31/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 564 kB/s | 90 kB 00:00 2024-02-10T06:05:19.394 INFO:teuthology.orchestra.run.smithi089.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-02-10T06:05:19.419 INFO:teuthology.orchestra.run.smithi089.stdout:(33/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-02-10T06:05:19.422 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:19.422 INFO:teuthology.orchestra.run.smithi089.stdout:Total 15 MB/s | 27 MB 00:01 2024-02-10T06:05:19.422 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:05:19.456 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:05:19.456 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:05:19.621 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:45 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:05:19.789 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:05:19.791 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 6.5 k 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.1 M 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 1.4 M 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 239 k 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.7 M 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 15 M 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-10T06:05:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-10T06:05:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout:Install 33 Packages 2024-02-10T06:05:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:19.795 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:05:19.795 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 27 M 2024-02-10T06:05:19.795 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 99 M 2024-02-10T06:05:19.795 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:05:19.796 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:05:20.011 INFO:teuthology.orchestra.run.smithi173.stdout:(1/33): ceph-17.2.7-505.g72ce7c1d.el9.x86_64.rp 38 kB/s | 6.5 kB 00:00 2024-02-10T06:05:20.295 INFO:teuthology.orchestra.run.smithi173.stdout:(2/33): ceph-mds-17.2.7-505.g72ce7c1d.el9.x86_6 4.6 MB/s | 2.1 MB 00:00 2024-02-10T06:05:20.370 INFO:teuthology.orchestra.run.smithi173.stdout:(3/33): ceph-mgr-17.2.7-505.g72ce7c1d.el9.x86_6 2.7 MB/s | 1.4 MB 00:00 2024-02-10T06:05:20.462 INFO:teuthology.orchestra.run.smithi173.stdout:(4/33): ceph-mgr-modules-core-17.2.7-505.g72ce7 2.5 MB/s | 239 kB 00:00 2024-02-10T06:05:20.546 INFO:teuthology.orchestra.run.smithi173.stdout:(5/33): ceph-mon-17.2.7-505.g72ce7c1d.el9.x86_6 6.9 MB/s | 3.7 MB 00:00 2024-02-10T06:05:20.613 INFO:teuthology.orchestra.run.smithi173.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 963 kB/s | 144 kB 00:00 2024-02-10T06:05:20.638 INFO:teuthology.orchestra.run.smithi173.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 348 kB/s | 32 kB 00:00 2024-02-10T06:05:20.663 INFO:teuthology.orchestra.run.smithi173.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2024-02-10T06:05:20.688 INFO:teuthology.orchestra.run.smithi173.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 768 kB/s | 39 kB 00:00 2024-02-10T06:05:20.722 INFO:teuthology.orchestra.run.smithi173.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-02-10T06:05:20.781 INFO:teuthology.orchestra.run.smithi173.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 13 MB/s | 1.1 MB 00:00 2024-02-10T06:05:20.806 INFO:teuthology.orchestra.run.smithi173.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-02-10T06:05:20.823 INFO:teuthology.orchestra.run.smithi173.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-02-10T06:05:20.848 INFO:teuthology.orchestra.run.smithi173.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 2.6 MB/s | 113 kB 00:00 2024-02-10T06:05:20.873 INFO:teuthology.orchestra.run.smithi173.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-02-10T06:05:20.957 INFO:teuthology.orchestra.run.smithi173.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 401 kB/s | 43 kB 00:00 2024-02-10T06:05:21.031 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:05:21.089 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-10T06:05:21.107 INFO:teuthology.orchestra.run.smithi173.stdout:(17/33): ceph-osd-17.2.7-505.g72ce7c1d.el9.x86_ 19 MB/s | 15 MB 00:00 2024-02-10T06:05:21.129 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-10T06:05:21.162 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-10T06:05:21.166 INFO:teuthology.orchestra.run.smithi173.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-02-10T06:05:21.191 INFO:teuthology.orchestra.run.smithi173.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-02-10T06:05:21.216 INFO:teuthology.orchestra.run.smithi173.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 501 kB/s | 172 kB 00:00 2024-02-10T06:05:21.242 INFO:teuthology.orchestra.run.smithi173.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 234 kB/s | 18 kB 00:00 2024-02-10T06:05:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-10T06:05:21.267 INFO:teuthology.orchestra.run.smithi173.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-02-10T06:05:21.292 INFO:teuthology.orchestra.run.smithi173.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-02-10T06:05:21.318 INFO:teuthology.orchestra.run.smithi173.stdout:(24/33): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-02-10T06:05:21.332 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-10T06:05:21.343 INFO:teuthology.orchestra.run.smithi173.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 610 kB/s | 46 kB 00:00 2024-02-10T06:05:21.360 INFO:teuthology.orchestra.run.smithi173.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-02-10T06:05:21.374 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-10T06:05:21.385 INFO:teuthology.orchestra.run.smithi173.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 393 kB/s | 16 kB 00:00 2024-02-10T06:05:21.410 INFO:teuthology.orchestra.run.smithi173.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-02-10T06:05:21.444 INFO:teuthology.orchestra.run.smithi173.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 612 kB/s | 36 kB 00:00 2024-02-10T06:05:21.445 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-10T06:05:21.461 INFO:teuthology.orchestra.run.smithi173.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-02-10T06:05:21.486 INFO:teuthology.orchestra.run.smithi173.stdout:(31/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-02-10T06:05:21.500 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-10T06:05:21.511 INFO:teuthology.orchestra.run.smithi173.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-02-10T06:05:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-10T06:05:21.570 INFO:teuthology.orchestra.run.smithi173.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-02-10T06:05:21.573 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:21.573 INFO:teuthology.orchestra.run.smithi173.stdout:Total 15 MB/s | 27 MB 00:01 2024-02-10T06:05:21.574 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:05:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-10T06:05:21.611 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:05:21.611 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:05:21.646 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-10T06:05:21.826 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-10T06:05:21.876 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-10T06:05:21.906 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-10T06:05:21.941 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-10T06:05:21.955 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:05:21.956 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:05:21.971 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-10T06:05:22.040 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-10T06:05:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-10T06:05:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-10T06:05:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-10T06:05:22.182 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-10T06:05:22.217 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-10T06:05:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-10T06:05:22.356 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-10T06:05:22.436 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-10T06:05:22.573 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-10T06:05:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-10T06:05:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 28/33 2024-02-10T06:05:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-10T06:05:22.896 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-10T06:05:22.896 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:22.896 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-10T06:05:22.897 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T06:05:22.897 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T06:05:22.897 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:23.193 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:05:23.256 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-10T06:05:23.279 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-10T06:05:23.312 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-10T06:05:23.403 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-10T06:05:23.482 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-10T06:05:23.523 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-10T06:05:23.595 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-10T06:05:23.649 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-10T06:05:23.688 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-10T06:05:23.737 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-10T06:05:23.798 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-10T06:05:23.984 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-10T06:05:24.057 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-10T06:05:24.089 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-10T06:05:24.123 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-10T06:05:24.153 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-10T06:05:24.230 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-10T06:05:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-10T06:05:24.256 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-10T06:05:24.256 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:24.256 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-10T06:05:24.256 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T06:05:24.256 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T06:05:24.256 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:24.263 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-10T06:05:24.297 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-10T06:05:24.363 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-10T06:05:24.389 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-10T06:05:24.426 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-10T06:05:24.504 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-10T06:05:24.538 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-10T06:05:24.609 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-10T06:05:24.631 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-10T06:05:24.653 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-10T06:05:24.653 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:24.654 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-10T06:05:24.654 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T06:05:24.654 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T06:05:24.654 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:24.745 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-10T06:05:24.871 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-10T06:05:24.873 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-10T06:05:24.895 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-10T06:05:24.895 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:24.895 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-10T06:05:24.895 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T06:05:24.896 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T06:05:24.896 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:24.963 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-10T06:05:25.037 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 28/33 2024-02-10T06:05:25.068 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-10T06:05:25.093 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-10T06:05:25.093 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:25.093 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-10T06:05:25.093 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T06:05:25.094 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T06:05:25.094 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:26.419 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-10T06:05:26.445 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-10T06:05:26.445 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:26.445 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-10T06:05:26.445 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T06:05:26.445 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T06:05:26.445 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:26.648 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-10T06:05:26.648 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/33 2024-02-10T06:05:26.648 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/33 2024-02-10T06:05:26.648 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/33 2024-02-10T06:05:26.648 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/33 2024-02-10T06:05:26.648 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/33 2024-02-10T06:05:26.648 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 6/33 2024-02-10T06:05:26.648 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-10T06:05:26.648 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-10T06:05:26.649 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-10T06:05:26.650 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-10T06:05:26.650 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-10T06:05:26.650 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-10T06:05:26.650 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-10T06:05:26.651 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-10T06:05:26.651 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-10T06:05:26.651 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-10T06:05:26.651 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-10T06:05:26.651 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-10T06:05:26.811 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-10T06:05:26.834 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-10T06:05:26.834 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:26.834 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-10T06:05:26.834 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T06:05:26.834 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T06:05:26.835 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:27.028 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-10T06:05:27.051 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-10T06:05:27.051 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:27.051 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-10T06:05:27.052 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T06:05:27.052 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T06:05:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-10T06:05:27.516 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-10T06:05:27.516 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:27.516 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:05:27.516 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-10T06:05:27.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply-3.11-14.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-10T06:05:27.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-10T06:05:27.519 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-10T06:05:27.519 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-10T06:05:27.519 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-10T06:05:27.519 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:27.519 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:05:27.841 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-base 2024-02-10T06:05:28.424 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:52 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:05:28.535 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:05:28.585 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:05:28.586 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-02-10T06:05:28.586 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:05:28.644 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install cephadm 2024-02-10T06:05:28.808 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 6/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-10T06:05:28.809 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-10T06:05:28.810 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-10T06:05:28.811 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-10T06:05:28.811 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-10T06:05:28.811 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-10T06:05:28.811 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-10T06:05:28.811 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-10T06:05:28.811 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-10T06:05:28.811 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-10T06:05:29.220 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:53 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:05:29.378 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:05:29.378 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 80 k 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 80 k 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 364 k 2024-02-10T06:05:29.379 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:05:29.599 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm-17.2.7-505.g72ce7c1d.el9.noarch.rpm 367 kB/s | 80 kB 00:00 2024-02-10T06:05:29.599 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:29.599 INFO:teuthology.orchestra.run.smithi089.stdout:Total 365 kB/s | 80 kB 00:00 2024-02-10T06:05:29.600 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:05:29.605 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:05:29.605 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:05:29.606 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:05:29.607 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:05:29.687 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:05:29.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-10T06:05:29.846 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:29.846 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:05:29.846 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:29.846 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:29.846 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:29.847 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:29.847 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:29.847 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:29.847 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-10T06:05:29.847 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-10T06:05:29.847 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-10T06:05:29.848 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-10T06:05:29.848 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-10T06:05:29.848 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-10T06:05:29.848 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-10T06:05:29.848 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-10T06:05:29.848 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-10T06:05:29.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-10T06:05:29.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-10T06:05:29.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-10T06:05:29.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-10T06:05:29.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-10T06:05:29.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-10T06:05:29.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-02-10T06:05:29.850 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-10T06:05:29.850 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-10T06:05:29.850 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-10T06:05:29.850 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-10T06:05:29.850 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-10T06:05:29.850 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-10T06:05:29.850 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-10T06:05:29.851 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-10T06:05:29.851 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-10T06:05:29.851 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-10T06:05:29.851 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-10T06:05:29.851 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:29.851 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:05:30.038 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:05:30.146 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:05:30.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-02-10T06:05:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:05:30.805 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:56 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:05:30.825 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:05:30.825 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:30.825 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:05:30.825 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:30.825 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:30.825 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:05:30.911 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:05:30.962 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:05:30.963 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-10T06:05:30.963 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:05:31.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-02-10T06:05:31.037 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T06:05:31.607 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:57 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:05:31.628 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:55 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:05:31.764 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:05:31.765 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:31.765 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-10T06:05:31.765 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:31.765 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:05:31.765 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 80 k 2024-02-10T06:05:31.765 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:31.765 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:05:31.766 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:31.766 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-10T06:05:31.766 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:31.766 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 80 k 2024-02-10T06:05:31.766 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 364 k 2024-02-10T06:05:31.766 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:05:31.788 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:05:31.788 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:31.788 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-02-10T06:05:31.788 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:31.789 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:05:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 137 k 2024-02-10T06:05:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:31.789 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:05:31.789 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:31.789 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-02-10T06:05:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:31.790 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 137 k 2024-02-10T06:05:31.790 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 379 k 2024-02-10T06:05:31.790 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:05:32.002 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-17.2.7-505.g72ce7c1d.el9.noarch.rpm 341 kB/s | 80 kB 00:00 2024-02-10T06:05:32.003 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:32.003 INFO:teuthology.orchestra.run.smithi173.stdout:Total 338 kB/s | 80 kB 00:00 2024-02-10T06:05:32.003 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:05:32.008 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:05:32.008 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:05:32.010 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:05:32.010 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:05:32.048 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-immutable-object-cache-17.2.7-505.g72ce7c1 530 kB/s | 137 kB 00:00 2024-02-10T06:05:32.049 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:32.049 INFO:teuthology.orchestra.run.smithi089.stdout:Total 528 kB/s | 137 kB 00:00 2024-02-10T06:05:32.049 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:05:32.056 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:05:32.056 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:05:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:05:32.108 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:05:32.108 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:05:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:05:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:05:32.339 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:05:32.340 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:32.340 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-10T06:05:32.340 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:32.524 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:05:32.623 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:05:33.013 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:05:33.307 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:05:33.308 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:33.308 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:05:33.308 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:33.308 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:33.308 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:05:33.355 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-10T06:05:33.355 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:33.356 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:05:33.356 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:33.356 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:33.356 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:05:33.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T06:05:33.565 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr 2024-02-10T06:05:34.144 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:00 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:05:34.154 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:58 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:05:34.260 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:05:34.301 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:05:34.302 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:34.302 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-02-10T06:05:34.302 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:34.303 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:05:34.303 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 137 k 2024-02-10T06:05:34.303 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:34.303 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:05:34.303 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:34.303 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-10T06:05:34.303 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:34.303 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 137 k 2024-02-10T06:05:34.303 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 379 k 2024-02-10T06:05:34.303 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:05:34.312 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:05:34.313 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-02-10T06:05:34.313 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:05:34.379 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T06:05:34.550 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-17.2.7-505.g72ce7c1 555 kB/s | 137 kB 00:00 2024-02-10T06:05:34.550 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:34.550 INFO:teuthology.orchestra.run.smithi173.stdout:Total 552 kB/s | 137 kB 00:00 2024-02-10T06:05:34.551 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:05:34.558 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:05:34.558 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:05:34.609 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:05:34.610 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:05:34.748 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:05:34.826 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:05:34.851 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:05:34.851 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:05:34.851 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-10T06:05:34.851 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:34.953 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:58 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:05:35.110 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:05:35.111 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-02-10T06:05:35.111 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:35.111 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:05:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 1.7 M 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 23 k 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 15 k 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout:Installing weak dependencies: 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout:Install 7 Packages 2024-02-10T06:05:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:35.113 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 2.0 M 2024-02-10T06:05:35.113 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 8.1 M 2024-02-10T06:05:35.113 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:05:35.859 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-10T06:05:35.859 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:35.859 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:05:35.860 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:05:35.860 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:35.860 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:05:36.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-02-10T06:05:36.536 INFO:teuthology.orchestra.run.smithi089.stdout:(1/7): ceph-prometheus-alerts-17.2.7-505.g72ce7 74 kB/s | 15 kB 00:00 2024-02-10T06:05:36.554 INFO:teuthology.orchestra.run.smithi089.stdout:(2/7): ceph-grafana-dashboards-17.2.7-505.g72ce 109 kB/s | 23 kB 00:00 2024-02-10T06:05:36.604 INFO:teuthology.orchestra.run.smithi089.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-02-10T06:05:36.646 INFO:teuthology.orchestra.run.smithi089.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 446 kB/s | 41 kB 00:00 2024-02-10T06:05:36.671 INFO:teuthology.orchestra.run.smithi089.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 460 kB/s | 31 kB 00:00 2024-02-10T06:05:36.709 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:02 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:05:36.729 INFO:teuthology.orchestra.run.smithi089.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-02-10T06:05:36.772 INFO:teuthology.orchestra.run.smithi089.stdout:(7/7): ceph-mgr-dashboard-17.2.7-505.g72ce7c1d. 3.9 MB/s | 1.7 MB 00:00 2024-02-10T06:05:36.773 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:36.773 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-02-10T06:05:36.774 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:05:36.782 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:05:36.782 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:05:36.816 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:05:36.850 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:05:36.851 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:05:36.867 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:05:36.868 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-10T06:05:36.868 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:05:36.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T06:05:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:05:37.214 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-10T06:05:37.248 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-10T06:05:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-10T06:05:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-10T06:05:37.364 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 5/7 2024-02-10T06:05:37.523 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:03 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:05:37.592 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 6/7 2024-02-10T06:05:37.617 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-10T06:05:37.682 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:05:37.682 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:37.683 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-10T06:05:37.683 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:37.683 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:05:37.683 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 1.7 M 2024-02-10T06:05:37.683 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-10T06:05:37.683 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 23 k 2024-02-10T06:05:37.683 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 15 k 2024-02-10T06:05:37.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-10T06:05:37.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-10T06:05:37.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-10T06:05:37.684 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-02-10T06:05:37.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-10T06:05:37.684 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:37.685 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:05:37.685 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:05:37.685 INFO:teuthology.orchestra.run.smithi173.stdout:Install 7 Packages 2024-02-10T06:05:37.685 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:37.686 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.0 M 2024-02-10T06:05:37.686 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 8.1 M 2024-02-10T06:05:37.687 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:05:37.938 INFO:teuthology.orchestra.run.smithi173.stdout:(1/7): ceph-prometheus-alerts-17.2.7-505.g72ce7 70 kB/s | 15 kB 00:00 2024-02-10T06:05:37.962 INFO:teuthology.orchestra.run.smithi173.stdout:(2/7): ceph-grafana-dashboards-17.2.7-505.g72ce 99 kB/s | 23 kB 00:00 2024-02-10T06:05:38.154 INFO:teuthology.orchestra.run.smithi173.stdout:(3/7): ceph-mgr-dashboard-17.2.7-505.g72ce7c1d. 4.0 MB/s | 1.7 MB 00:00 2024-02-10T06:05:38.161 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-10T06:05:38.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 1/7 2024-02-10T06:05:38.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2/7 2024-02-10T06:05:38.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 3/7 2024-02-10T06:05:38.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-10T06:05:38.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-10T06:05:38.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-10T06:05:38.346 INFO:teuthology.orchestra.run.smithi173.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 22 kB/s | 9.0 kB 00:00 2024-02-10T06:05:38.464 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-10T06:05:38.464 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:38.464 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:05:38.464 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:38.464 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:38.464 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:38.464 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-10T06:05:38.465 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-10T06:05:38.465 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-10T06:05:38.465 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-10T06:05:38.465 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:38.465 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:05:38.523 INFO:teuthology.orchestra.run.smithi173.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 73 kB/s | 41 kB 00:00 2024-02-10T06:05:38.613 INFO:teuthology.orchestra.run.smithi173.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 67 kB/s | 31 kB 00:00 2024-02-10T06:05:38.731 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T06:05:38.789 INFO:teuthology.orchestra.run.smithi173.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 427 kB/s | 188 kB 00:00 2024-02-10T06:05:38.793 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:38.793 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-02-10T06:05:38.793 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:05:38.800 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:05:38.800 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:05:38.869 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:05:38.869 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:05:39.195 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:05:39.248 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-10T06:05:39.282 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-10T06:05:39.302 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:03 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:05:39.328 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-10T06:05:39.360 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-10T06:05:39.389 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 5/7 2024-02-10T06:05:39.463 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:05:39.464 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================== 2024-02-10T06:05:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-02-10T06:05:39.464 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================== 2024-02-10T06:05:39.464 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:05:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 7.4 M 2024-02-10T06:05:39.464 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-02-10T06:05:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-10T06:05:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-10T06:05:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-10T06:05:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================== 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout:Install 12 Packages 2024-02-10T06:05:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:39.466 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 37 M 2024-02-10T06:05:39.466 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 209 M 2024-02-10T06:05:39.466 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:05:39.616 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 6/7 2024-02-10T06:05:39.650 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-10T06:05:39.671 INFO:teuthology.orchestra.run.smithi089.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2024-02-10T06:05:39.705 INFO:teuthology.orchestra.run.smithi089.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 524 kB/s | 17 kB 00:00 2024-02-10T06:05:39.747 INFO:teuthology.orchestra.run.smithi089.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 846 kB/s | 34 kB 00:00 2024-02-10T06:05:39.806 INFO:teuthology.orchestra.run.smithi089.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-02-10T06:05:40.006 INFO:teuthology.orchestra.run.smithi089.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-02-10T06:05:40.073 INFO:teuthology.orchestra.run.smithi089.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-02-10T06:05:40.098 INFO:teuthology.orchestra.run.smithi089.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-02-10T06:05:40.196 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-10T06:05:40.196 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 1/7 2024-02-10T06:05:40.196 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2/7 2024-02-10T06:05:40.196 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 3/7 2024-02-10T06:05:40.197 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-10T06:05:40.197 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-10T06:05:40.197 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-10T06:05:40.299 INFO:teuthology.orchestra.run.smithi089.stdout:(8/12): ceph-mgr-diskprediction-local-17.2.7-50 9.6 MB/s | 7.4 MB 00:00 2024-02-10T06:05:40.332 INFO:teuthology.orchestra.run.smithi089.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 802 kB 00:00 2024-02-10T06:05:40.499 INFO:teuthology.orchestra.run.smithi089.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-02-10T06:05:40.524 INFO:teuthology.orchestra.run.smithi089.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 874 kB/s | 194 kB 00:00 2024-02-10T06:05:40.527 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-10T06:05:40.527 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:40.527 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:05:40.527 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:40.527 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:40.527 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:40.527 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-10T06:05:40.527 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-10T06:05:40.527 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-10T06:05:40.527 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-10T06:05:40.528 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:40.528 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:05:40.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T06:05:40.983 INFO:teuthology.orchestra.run.smithi089.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 378 kB/s | 245 kB 00:00 2024-02-10T06:05:40.985 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:40.986 INFO:teuthology.orchestra.run.smithi089.stdout:Total 24 MB/s | 37 MB 00:01 2024-02-10T06:05:41.007 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:05:41.032 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:05:41.032 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:05:41.351 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:07 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:05:41.461 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:05:41.461 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:05:41.511 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout:=========================================================================================== 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout:=========================================================================================== 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 7.4 M 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-10T06:05:41.513 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-10T06:05:41.514 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-10T06:05:41.514 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-10T06:05:41.514 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-10T06:05:41.514 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-10T06:05:41.514 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-10T06:05:41.514 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:41.514 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:05:41.514 INFO:teuthology.orchestra.run.smithi173.stdout:=========================================================================================== 2024-02-10T06:05:41.514 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-02-10T06:05:41.514 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:41.515 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 37 M 2024-02-10T06:05:41.515 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 209 M 2024-02-10T06:05:41.515 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:05:41.687 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 242 kB/s | 32 kB 00:00 2024-02-10T06:05:41.721 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 539 kB/s | 17 kB 00:00 2024-02-10T06:05:41.762 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 855 kB/s | 34 kB 00:00 2024-02-10T06:05:41.813 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-02-10T06:05:41.996 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-02-10T06:05:42.063 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-02-10T06:05:42.080 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-02-10T06:05:42.151 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:05:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-10T06:05:42.272 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 802 kB 00:00 2024-02-10T06:05:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-10T06:05:42.372 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-10T06:05:42.416 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-10T06:05:42.423 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-02-10T06:05:42.457 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-02-10T06:05:42.584 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T06:05:42.590 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-02-10T06:05:42.632 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T06:05:42.673 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-5 6.6 MB/s | 7.4 MB 00:01 2024-02-10T06:05:42.676 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:42.676 INFO:teuthology.orchestra.run.smithi173.stdout:Total 32 MB/s | 37 MB 00:01 2024-02-10T06:05:42.694 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:05:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-10T06:05:42.718 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:05:42.718 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:05:43.167 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:05:43.168 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:05:43.351 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-10T06:05:43.477 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-10T06:05:43.875 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:05:43.940 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-10T06:05:43.969 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-10T06:05:44.061 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-10T06:05:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-10T06:05:44.266 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T06:05:44.305 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T06:05:44.389 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-10T06:05:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-10T06:05:45.049 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-10T06:05:45.165 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-10T06:05:45.766 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-10T06:05:45.800 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-10T06:05:46.391 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-10T06:05:47.433 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-10T06:05:47.462 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-10T06:05:48.299 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-10T06:05:48.299 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 1/12 2024-02-10T06:05:48.299 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-10T06:05:48.299 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-10T06:05:48.299 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-10T06:05:48.299 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T06:05:48.299 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T06:05:48.299 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-10T06:05:48.299 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-10T06:05:48.300 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-10T06:05:48.300 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-10T06:05:48.300 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-10T06:05:48.769 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-10T06:05:48.770 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-10T06:05:48.770 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-10T06:05:48.770 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:48.770 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:05:49.181 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-rook 2024-02-10T06:05:49.791 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:13 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:05:49.960 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 47 k 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-10T06:05:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-10T06:05:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-10T06:05:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-10T06:05:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-10T06:05:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-10T06:05:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-10T06:05:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-10T06:05:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:49.963 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:05:49.963 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2024-02-10T06:05:49.963 INFO:teuthology.orchestra.run.smithi089.stdout:Install 13 Packages 2024-02-10T06:05:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:49.964 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 2.0 M 2024-02-10T06:05:49.964 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 26 M 2024-02-10T06:05:49.964 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:05:50.028 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-10T06:05:50.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 1/12 2024-02-10T06:05:50.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-10T06:05:50.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-10T06:05:50.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-10T06:05:50.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T06:05:50.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T06:05:50.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-10T06:05:50.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-10T06:05:50.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-10T06:05:50.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-10T06:05:50.029 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-10T06:05:50.169 INFO:teuthology.orchestra.run.smithi089.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 184 kB/s | 24 kB 00:00 2024-02-10T06:05:50.186 INFO:teuthology.orchestra.run.smithi089.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-02-10T06:05:50.236 INFO:teuthology.orchestra.run.smithi089.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-02-10T06:05:50.261 INFO:teuthology.orchestra.run.smithi089.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-02-10T06:05:50.287 INFO:teuthology.orchestra.run.smithi089.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-10T06:05:50.312 INFO:teuthology.orchestra.run.smithi089.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-02-10T06:05:50.337 INFO:teuthology.orchestra.run.smithi089.stdout:(7/13): ceph-mgr-rook-17.2.7-505.g72ce7c1d.el9. 158 kB/s | 47 kB 00:00 2024-02-10T06:05:50.446 INFO:teuthology.orchestra.run.smithi089.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2024-02-10T06:05:50.480 INFO:teuthology.orchestra.run.smithi089.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-02-10T06:05:50.511 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-10T06:05:50.512 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-10T06:05:50.513 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:50.513 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:05:50.622 INFO:teuthology.orchestra.run.smithi089.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 714 kB/s | 203 kB 00:00 2024-02-10T06:05:50.647 INFO:teuthology.orchestra.run.smithi089.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-02-10T06:05:50.815 INFO:teuthology.orchestra.run.smithi089.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-02-10T06:05:50.839 INFO:teuthology.orchestra.run.smithi089.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 413 kB/s | 90 kB 00:00 2024-02-10T06:05:50.842 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:50.842 INFO:teuthology.orchestra.run.smithi089.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-02-10T06:05:50.842 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:05:50.855 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:05:50.855 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:05:50.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-rook 2024-02-10T06:05:50.952 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:05:50.956 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:05:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:05:51.514 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:17 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:05:51.644 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-10T06:05:51.682 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:05:51.683 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-02-10T06:05:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-10T06:05:51.683 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 47 k 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-10T06:05:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-10T06:05:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-10T06:05:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:51.685 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:05:51.685 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-02-10T06:05:51.685 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-02-10T06:05:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:51.685 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.0 M 2024-02-10T06:05:51.685 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-02-10T06:05:51.686 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:05:51.698 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-10T06:05:51.754 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-10T06:05:51.792 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-10T06:05:51.834 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-10T06:05:51.927 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-10T06:05:52.043 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-10T06:05:52.110 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-10T06:05:52.154 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 142 kB/s | 18 kB 00:00 2024-02-10T06:05:52.179 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 159 kB/s | 24 kB 00:00 2024-02-10T06:05:52.230 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-02-10T06:05:52.255 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-02-10T06:05:52.280 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-10T06:05:52.297 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-02-10T06:05:52.314 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): ceph-mgr-rook-17.2.7-505.g72ce7c1d.el9. 165 kB/s | 47 kB 00:00 2024-02-10T06:05:52.532 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-02-10T06:05:52.548 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-02-10T06:05:52.732 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-02-10T06:05:52.807 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 411 kB/s | 203 kB 00:00 2024-02-10T06:05:52.829 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-10T06:05:52.874 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-10T06:05:52.899 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 538 kB/s | 90 kB 00:00 2024-02-10T06:05:52.908 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-10T06:05:52.954 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-10T06:05:52.978 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-10T06:05:53.066 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-02-10T06:05:53.068 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:53.068 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-02-10T06:05:53.068 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:05:53.081 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:05:53.081 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:05:53.180 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:05:53.180 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:05:53.690 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 1/13 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-10T06:05:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-10T06:05:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-10T06:05:53.863 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-10T06:05:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-10T06:05:53.980 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-10T06:05:54.018 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-10T06:05:54.060 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-10T06:05:54.151 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-10T06:05:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-10T06:05:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:54.233 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:05:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-10T06:05:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-10T06:05:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-10T06:05:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-10T06:05:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-10T06:05:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-10T06:05:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-10T06:05:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-10T06:05:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-10T06:05:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-10T06:05:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-10T06:05:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-10T06:05:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:54.234 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:05:54.261 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-10T06:05:54.310 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-10T06:05:54.484 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T06:05:55.030 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-10T06:05:55.067 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:19 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:05:55.074 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-10T06:05:55.108 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-10T06:05:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-10T06:05:55.178 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-10T06:05:55.223 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:05:55.224 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2024-02-10T06:05:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-02-10T06:05:55.224 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2024-02-10T06:05:55.224 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 121 k 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-02-10T06:05:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:55.226 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 6.8 M 2024-02-10T06:05:55.226 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 33 M 2024-02-10T06:05:55.226 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:05:55.527 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 227 kB 00:00 2024-02-10T06:05:55.660 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-02-10T06:05:55.719 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): ceph-mgr-cephadm-17.2.7-505.g72ce7c1d.el 341 kB/s | 121 kB 00:00 2024-02-10T06:05:55.964 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-10T06:05:55.964 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 1/13 2024-02-10T06:05:55.964 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-10T06:05:55.964 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-10T06:05:55.964 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-10T06:05:55.964 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-10T06:05:55.964 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-10T06:05:55.965 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-10T06:05:55.965 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-10T06:05:55.965 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-10T06:05:55.965 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-10T06:05:55.965 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-10T06:05:55.965 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-10T06:05:56.036 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 154 kB/s | 58 kB 00:00 2024-02-10T06:05:56.103 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 133 kB/s | 51 kB 00:00 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-10T06:05:56.359 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-10T06:05:56.360 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-10T06:05:56.360 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-10T06:05:56.360 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-10T06:05:56.360 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-10T06:05:56.360 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:56.360 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:05:56.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T06:05:56.936 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 389 kB/s | 548 kB 00:01 2024-02-10T06:05:56.938 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:56.939 INFO:teuthology.orchestra.run.smithi089.stdout:Total 3.9 MB/s | 6.8 MB 00:01 2024-02-10T06:05:56.939 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:05:56.949 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:05:56.949 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:05:57.043 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:05:57.045 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:05:57.226 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:23 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:05:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:05:57.386 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:05:57.386 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 121 k 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-02-10T06:05:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:05:57.388 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 6.8 M 2024-02-10T06:05:57.389 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 33 M 2024-02-10T06:05:57.389 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:05:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-10T06:05:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-10T06:05:57.601 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 227 kB 00:00 2024-02-10T06:05:57.684 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): ceph-mgr-cephadm-17.2.7-505.g72ce7c1d.el 479 kB/s | 121 kB 00:00 2024-02-10T06:05:57.801 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-02-10T06:05:58.087 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-10T06:05:58.166 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-10T06:05:58.220 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-10T06:05:58.259 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-10T06:05:58.277 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 102 kB/s | 58 kB 00:00 2024-02-10T06:05:58.352 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 97 kB/s | 51 kB 00:00 2024-02-10T06:05:58.702 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 498 kB/s | 548 kB 00:01 2024-02-10T06:05:58.706 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:05:58.706 INFO:teuthology.orchestra.run.smithi173.stdout:Total 5.1 MB/s | 6.8 MB 00:01 2024-02-10T06:05:58.706 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:05:58.716 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:05:58.716 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:05:58.806 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:05:58.808 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:05:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-10T06:05:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/6 2024-02-10T06:05:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-10T06:05:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-10T06:05:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-10T06:05:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-10T06:05:59.108 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:05:59.275 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-10T06:05:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-10T06:05:59.421 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-10T06:05:59.421 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:59.421 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:05:59.421 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:05:59.421 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-10T06:05:59.421 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-10T06:05:59.421 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-10T06:05:59.421 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-10T06:05:59.421 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-10T06:05:59.422 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:05:59.422 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:05:59.671 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-fuse 2024-02-10T06:05:59.851 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-10T06:05:59.922 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-10T06:05:59.986 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-10T06:06:00.015 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-10T06:06:00.259 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:24 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:00.417 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:00.417 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:00.417 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-02-10T06:06:00.417 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:00.418 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:06:00.418 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 812 k 2024-02-10T06:06:00.418 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-02-10T06:06:00.418 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-10T06:06:00.418 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:00.418 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:06:00.418 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:00.418 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2024-02-10T06:06:00.418 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:00.419 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 891 k 2024-02-10T06:06:00.419 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 2.6 M 2024-02-10T06:06:00.419 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:06:00.672 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 310 kB/s | 79 kB 00:00 2024-02-10T06:06:00.876 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-10T06:06:00.876 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/6 2024-02-10T06:06:00.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-10T06:06:00.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-10T06:06:00.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-10T06:06:00.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-10T06:06:00.906 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): ceph-fuse-17.2.7-505.g72ce7c1d.el9.x86_6 1.6 MB/s | 812 kB 00:00 2024-02-10T06:06:00.908 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:00.908 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.8 MB/s | 891 kB 00:00 2024-02-10T06:06:00.908 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:06:00.929 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:06:00.929 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:06:01.003 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:06:01.003 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:06:01.193 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:06:01.199 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-10T06:06:01.200 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:01.200 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:06:01.200 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-10T06:06:01.200 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-10T06:06:01.200 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-10T06:06:01.200 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-10T06:06:01.200 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-10T06:06:01.200 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-10T06:06:01.200 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:01.200 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:01.317 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-10T06:06:01.380 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-10T06:06:01.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-02-10T06:06:02.084 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:28 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:02.109 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-10T06:06:02.109 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/2 2024-02-10T06:06:02.243 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:02.245 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:02.245 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-10T06:06:02.245 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:02.245 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:06:02.245 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 812 k 2024-02-10T06:06:02.245 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-10T06:06:02.245 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-10T06:06:02.245 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:02.245 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:06:02.246 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:02.246 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-02-10T06:06:02.246 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:02.246 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 891 k 2024-02-10T06:06:02.246 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 2.6 M 2024-02-10T06:06:02.246 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:06:02.378 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 592 kB/s | 79 kB 00:00 2024-02-10T06:06:02.429 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-10T06:06:02.429 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:02.429 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:06:02.429 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-10T06:06:02.429 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:02.429 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:02.612 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-fuse-17.2.7-505.g72ce7c1d.el9.x86_6 2.2 MB/s | 812 kB 00:00 2024-02-10T06:06:02.613 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:02.614 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.4 MB/s | 891 kB 00:00 2024-02-10T06:06:02.614 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:06:02.634 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:06:02.634 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:06:02.679 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados-devel 2024-02-10T06:06:02.708 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:06:02.708 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:06:02.888 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:06:03.032 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-10T06:06:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-10T06:06:03.273 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:27 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:03.430 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:03.431 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-02-10T06:06:03.431 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:03.431 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:06:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 126 k 2024-02-10T06:06:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:03.431 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:06:03.431 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:03.431 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-02-10T06:06:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:03.432 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 126 k 2024-02-10T06:06:03.432 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 448 k 2024-02-10T06:06:03.432 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:06:03.654 INFO:teuthology.orchestra.run.smithi089.stdout:librados-devel-17.2.7-505.g72ce7c1d.el9.x86_64. 569 kB/s | 126 kB 00:00 2024-02-10T06:06:03.654 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:03.655 INFO:teuthology.orchestra.run.smithi089.stdout:Total 566 kB/s | 126 kB 00:00 2024-02-10T06:06:03.655 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:06:03.660 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:06:03.660 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:06:03.715 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:06:03.715 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:06:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-10T06:06:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/2 2024-02-10T06:06:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:06:03.977 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:04.124 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-10T06:06:04.124 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:04.124 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:06:04.124 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-10T06:06:04.124 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:04.124 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:04.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-02-10T06:06:04.414 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:04.701 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:04.701 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:04.701 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:06:04.701 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:06:04.701 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:04.702 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:04.947 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs2 2024-02-10T06:06:04.999 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:30 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:05.158 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:05.158 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:05.158 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-02-10T06:06:05.158 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:05.159 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:06:05.159 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 126 k 2024-02-10T06:06:05.159 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:05.159 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:06:05.159 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:05.159 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-10T06:06:05.159 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:05.159 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 126 k 2024-02-10T06:06:05.159 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 448 k 2024-02-10T06:06:05.159 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:06:05.408 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-17.2.7-505.g72ce7c1d.el9.x86_64. 509 kB/s | 126 kB 00:00 2024-02-10T06:06:05.408 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:05.409 INFO:teuthology.orchestra.run.smithi173.stdout:Total 506 kB/s | 126 kB 00:00 2024-02-10T06:06:05.409 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:06:05.414 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:06:05.414 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:06:05.469 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:06:05.469 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:06:05.540 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:29 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:05.624 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:06:05.650 INFO:teuthology.orchestra.run.smithi089.stdout:Package libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:05.703 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:05.704 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-02-10T06:06:05.704 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:05.722 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:05.767 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs-devel 2024-02-10T06:06:06.189 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:06.350 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:30 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:06.507 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:06.507 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:06.508 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-02-10T06:06:06.508 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:06.508 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:06:06.508 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-10T06:06:06.508 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:06.508 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:06:06.508 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:06.508 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-02-10T06:06:06.508 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:06.508 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 25 k 2024-02-10T06:06:06.508 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 110 k 2024-02-10T06:06:06.509 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:06:06.515 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:06.515 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:06.515 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:06:06.515 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:06:06.516 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:06.516 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:06.685 INFO:teuthology.orchestra.run.smithi089.stdout:libcephfs-devel-17.2.7-505.g72ce7c1d.el9.x86_64 142 kB/s | 25 kB 00:00 2024-02-10T06:06:06.685 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:06.685 INFO:teuthology.orchestra.run.smithi089.stdout:Total 140 kB/s | 25 kB 00:00 2024-02-10T06:06:06.686 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:06:06.689 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:06:06.689 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:06:06.691 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:06:06.691 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:06:06.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-02-10T06:06:06.777 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:06:06.890 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:07.272 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:07.368 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:33 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:07.478 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:07.532 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:07.533 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-10T06:06:07.534 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:07.543 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:07.543 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:07.543 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:06:07.543 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:06:07.543 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:07.543 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:07.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-02-10T06:06:07.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados2 2024-02-10T06:06:08.211 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:34 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:08.305 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:32 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:08.372 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:08.373 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:08.373 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-02-10T06:06:08.373 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:08.373 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:06:08.373 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-10T06:06:08.373 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:08.373 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:06:08.374 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:08.374 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-10T06:06:08.374 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:08.374 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 25 k 2024-02-10T06:06:08.374 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 110 k 2024-02-10T06:06:08.374 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:06:08.413 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:08.464 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:08.465 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-02-10T06:06:08.465 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:08.527 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librbd1 2024-02-10T06:06:08.561 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-17.2.7-505.g72ce7c1d.el9.x86_64 133 kB/s | 25 kB 00:00 2024-02-10T06:06:08.562 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:08.562 INFO:teuthology.orchestra.run.smithi173.stdout:Total 132 kB/s | 25 kB 00:00 2024-02-10T06:06:08.562 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:06:08.566 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:06:08.566 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:06:08.567 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:06:08.568 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:06:08.654 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:06:08.792 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:09.114 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:33 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:09.184 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:09.221 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:09.275 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:09.276 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-02-10T06:06:09.276 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:09.341 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rados 2024-02-10T06:06:09.462 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:09.463 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:09.463 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:06:09.463 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:06:09.463 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:09.463 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:09.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-02-10T06:06:09.922 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:33 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:10.028 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:10.081 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:10.082 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-02-10T06:06:10.083 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:10.147 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rgw 2024-02-10T06:06:10.259 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:36 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:10.366 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:10.419 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:10.420 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-10T06:06:10.420 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:10.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-02-10T06:06:10.725 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:34 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:10.831 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:10.884 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:10.885 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-02-10T06:06:10.885 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:10.949 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-cephfs 2024-02-10T06:06:11.088 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:37 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:11.194 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:11.246 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:11.247 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-10T06:06:11.247 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:11.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-02-10T06:06:11.536 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:35 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:11.643 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:11.695 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:11.696 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-02-10T06:06:11.696 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:11.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rbd 2024-02-10T06:06:11.909 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:37 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:12.015 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:12.067 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:12.068 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-10T06:06:12.069 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:12.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-02-10T06:06:12.344 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:36 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:12.452 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:12.505 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:12.508 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-02-10T06:06:12.508 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:12.570 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-fuse 2024-02-10T06:06:12.738 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:38 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:12.846 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:12.899 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:12.900 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-10T06:06:12.900 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:12.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-02-10T06:06:13.156 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:37 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:13.314 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:13.315 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:13.315 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-02-10T06:06:13.315 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:13.315 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:06:13.315 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 81 k 2024-02-10T06:06:13.316 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:13.316 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:06:13.316 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:13.316 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-02-10T06:06:13.316 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:13.316 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 81 k 2024-02-10T06:06:13.316 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 222 k 2024-02-10T06:06:13.316 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:06:13.546 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-fuse-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 353 kB/s | 81 kB 00:00 2024-02-10T06:06:13.547 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:13.547 INFO:teuthology.orchestra.run.smithi089.stdout:Total 352 kB/s | 81 kB 00:00 2024-02-10T06:06:13.547 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:06:13.554 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:06:13.554 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:06:13.565 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:39 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:13.610 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:06:13.610 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:06:13.671 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:13.723 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:13.724 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-10T06:06:13.724 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:13.742 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:06:13.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-02-10T06:06:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:14.218 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:14.388 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:40 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:14.494 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-10T06:06:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:14.509 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:06:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:06:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:14.509 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:14.547 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:14.548 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-10T06:06:14.548 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:14.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-02-10T06:06:14.742 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-mirror 2024-02-10T06:06:15.221 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:41 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:15.336 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:39 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:15.379 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:15.380 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:15.380 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-10T06:06:15.381 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:15.381 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:06:15.381 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 81 k 2024-02-10T06:06:15.381 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:15.381 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:06:15.381 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:15.381 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-10T06:06:15.381 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:15.381 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 81 k 2024-02-10T06:06:15.381 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 222 k 2024-02-10T06:06:15.381 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:06:15.500 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:15.501 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:15.501 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-02-10T06:06:15.501 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:15.501 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:06:15.501 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-10T06:06:15.501 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:15.502 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:06:15.502 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:15.502 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-02-10T06:06:15.502 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:15.502 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 2.9 M 2024-02-10T06:06:15.502 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 12 M 2024-02-10T06:06:15.502 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:06:15.639 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 316 kB/s | 81 kB 00:00 2024-02-10T06:06:15.639 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:15.640 INFO:teuthology.orchestra.run.smithi173.stdout:Total 314 kB/s | 81 kB 00:00 2024-02-10T06:06:15.640 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:06:15.646 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:06:15.647 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:06:15.701 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:06:15.701 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:06:15.835 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:06:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:16.211 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-mirror-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 4.1 MB/s | 2.9 MB 00:00 2024-02-10T06:06:16.211 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:16.212 INFO:teuthology.orchestra.run.smithi089.stdout:Total 4.1 MB/s | 2.9 MB 00:00 2024-02-10T06:06:16.212 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:06:16.221 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:06:16.221 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:06:16.298 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:06:16.298 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:06:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:16.712 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:16.712 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:16.712 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:06:16.712 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:06:16.712 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:16.712 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:16.714 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:06:16.787 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:16.811 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:16.811 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:06:16.811 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-10T06:06:16.812 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T06:06:16.812 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T06:06:16.812 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:16.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-02-10T06:06:17.559 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:43 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:17.722 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.9 M 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 12 M 2024-02-10T06:06:17.723 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:06:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:17.999 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:06:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:06:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:17.999 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:18.159 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-nbd 2024-02-10T06:06:18.327 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 4.9 MB/s | 2.9 MB 00:00 2024-02-10T06:06:18.327 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:18.328 INFO:teuthology.orchestra.run.smithi173.stdout:Total 4.9 MB/s | 2.9 MB 00:00 2024-02-10T06:06:18.328 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:06:18.337 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:06:18.337 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:06:18.413 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:06:18.413 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:06:18.755 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:42 ago on Sat 10 Feb 2024 06:04:36 AM UTC. 2024-02-10T06:06:18.830 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:06:18.894 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:18.911 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-10T06:06:18.912 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:18.912 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-02-10T06:06:18.912 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:18.912 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-10T06:06:18.912 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 164 k 2024-02-10T06:06:18.912 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:18.912 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-10T06:06:18.912 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-02-10T06:06:18.913 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-02-10T06:06:18.913 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:18.913 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 164 k 2024-02-10T06:06:18.913 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 477 k 2024-02-10T06:06:18.913 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-10T06:06:18.918 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:18.918 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T06:06:18.918 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-10T06:06:18.919 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T06:06:18.919 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T06:06:18.919 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:19.165 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-nbd-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 653 kB/s | 164 kB 00:00 2024-02-10T06:06:19.166 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:19.166 INFO:teuthology.orchestra.run.smithi089.stdout:Total 650 kB/s | 164 kB 00:00 2024-02-10T06:06:19.166 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-10T06:06:19.174 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-10T06:06:19.174 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-10T06:06:19.229 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-10T06:06:19.230 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-10T06:06:19.384 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-10T06:06:19.496 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:19.940 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:20.037 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:06:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:06:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:20.037 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:20.253 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:20.253 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:20.254 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-10T06:06:20.254 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:06:20.254 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:06:20.254 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-10T06:06:20.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-02-10T06:06:20.498 DEBUG:teuthology.parallel:result is None 2024-02-10T06:06:20.879 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:46 ago on Sat 10 Feb 2024 06:04:34 AM UTC. 2024-02-10T06:06:21.038 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-10T06:06:21.039 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-10T06:06:21.039 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:21.039 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-10T06:06:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 164 k 2024-02-10T06:06:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:21.040 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-10T06:06:21.040 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-10T06:06:21.040 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-10T06:06:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:21.040 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 164 k 2024-02-10T06:06:21.040 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 477 k 2024-02-10T06:06:21.040 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-10T06:06:21.414 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 440 kB/s | 164 kB 00:00 2024-02-10T06:06:21.414 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:06:21.415 INFO:teuthology.orchestra.run.smithi173.stdout:Total 439 kB/s | 164 kB 00:00 2024-02-10T06:06:21.415 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-10T06:06:21.422 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-10T06:06:21.422 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-10T06:06:21.477 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-10T06:06:21.477 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-10T06:06:21.626 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-10T06:06:21.744 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:22.264 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:22.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-10T06:06:22.588 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:22.588 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-10T06:06:22.588 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-10T06:06:22.589 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:06:22.589 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-10T06:06:22.824 DEBUG:teuthology.parallel:result is None 2024-02-10T06:06:22.825 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:06:22.825 INFO:teuthology.packaging:ref: None 2024-02-10T06:06:22.825 INFO:teuthology.packaging:tag: None 2024-02-10T06:06:22.825 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:06:22.825 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:06:22.825 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-10T06:06:23.072 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T06:06:23.098 INFO:teuthology.orchestra.run.smithi089.stdout:17.2.7-505.g72ce7c1d.el9 2024-02-10T06:06:23.098 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-505.g72ce7c1d.el9 2024-02-10T06:06:23.098 INFO:teuthology.task.install:The correct ceph version 17.2.7-505.g72ce7c1d is installed. 2024-02-10T06:06:23.100 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T06:06:23.100 INFO:teuthology.packaging:ref: None 2024-02-10T06:06:23.100 INFO:teuthology.packaging:tag: None 2024-02-10T06:06:23.100 INFO:teuthology.packaging:branch: quincy 2024-02-10T06:06:23.100 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:06:23.101 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-10T06:06:23.344 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T06:06:23.365 INFO:teuthology.orchestra.run.smithi173.stdout:17.2.7-505.g72ce7c1d.el9 2024-02-10T06:06:23.365 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-505.g72ce7c1d.el9 2024-02-10T06:06:23.365 INFO:teuthology.task.install:The correct ceph version 17.2.7-505.g72ce7c1d is installed. 2024-02-10T06:06:23.367 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-10T06:06:23.367 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:06:23.368 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T06:06:23.400 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:06:23.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T06:06:23.434 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-10T06:06:23.434 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:06:23.434 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T06:06:23.473 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T06:06:23.543 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:06:23.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T06:06:23.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T06:06:23.637 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-10T06:06:23.638 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:06:23.638 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T06:06:23.668 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T06:06:23.739 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:06:23.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T06:06:23.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T06:06:23.832 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-10T06:06:23.833 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:06:23.833 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T06:06:23.863 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T06:06:23.933 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:06:23.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T06:06:23.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T06:06:24.026 INFO:teuthology.run_tasks:Running task print... 2024-02-10T06:06:24.036 INFO:teuthology.task.print:**** done install task... 2024-02-10T06:06:24.036 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T06:06:24.150 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T06:06:24.150 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-02-10T06:06:24.150 INFO:tasks.cephadm:Cluster fsid is 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:06:24.151 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T06:06:24.151 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-10T06:06:24.151 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi089': '172.21.15.89', 'mon.smithi173': '172.21.15.173'} 2024-02-10T06:06:24.151 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T06:06:24.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname $(hostname -s) 2024-02-10T06:06:24.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-02-10T06:06:24.209 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-02-10T06:06:24.209 DEBUG:teuthology.orchestra.run.smithi089:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T06:06:24.450 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 10 06:06 /home/ubuntu/cephtest/cephadm 2024-02-10T06:06:24.450 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T06:06:24.652 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 10 06:06 /home/ubuntu/cephtest/cephadm 2024-02-10T06:06:24.652 DEBUG:teuthology.orchestra.run.smithi089:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T06:06:24.671 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T06:06:24.695 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-02-10T06:06:24.695 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-10T06:06:24.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-10T06:06:25.570 INFO:teuthology.orchestra.run.smithi089.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-10T06:06:25.641 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-10T06:07:19.606 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-10T06:07:19.606 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_version": "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)", 2024-02-10T06:07:19.606 INFO:teuthology.orchestra.run.smithi089.stdout: "image_id": "a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5", 2024-02-10T06:07:19.607 INFO:teuthology.orchestra.run.smithi089.stdout: "repo_digests": [ 2024-02-10T06:07:19.607 INFO:teuthology.orchestra.run.smithi089.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc" 2024-02-10T06:07:19.607 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-02-10T06:07:19.607 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-10T06:07:24.378 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-02-10T06:07:24.378 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)", 2024-02-10T06:07:24.378 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5", 2024-02-10T06:07:24.378 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-02-10T06:07:24.378 INFO:teuthology.orchestra.run.smithi173.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc" 2024-02-10T06:07:24.378 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-02-10T06:07:24.378 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-02-10T06:07:24.402 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph 2024-02-10T06:07:24.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-02-10T06:07:24.467 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /etc/ceph 2024-02-10T06:07:24.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-02-10T06:07:24.534 INFO:tasks.cephadm:Writing seed config... 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T06:07:24.536 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-10T06:07:24.537 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T06:07:24.538 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-10T06:07:24.539 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:07:24.539 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T06:07:24.565 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 84e5b1ba-c7da-11ee-95b8-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-10T06:07:24.565 DEBUG:teuthology.orchestra.run.smithi089:mon.smithi089> sudo journalctl -f -n 0 -u ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mon.smithi089.service 2024-02-10T06:07:24.607 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T06:07:24.609 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.89 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:07:24.815 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-10T06:07:24.815 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '84e5b1ba-c7da-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.89', '--skip-admin-label'] 2024-02-10T06:07:24.840 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T06:07:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-10T06:07:24.840 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying podman|docker is present... 2024-02-10T06:07:24.864 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T06:07:24.864 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying lvm2 is present... 2024-02-10T06:07:24.864 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying time synchronization is in place... 2024-02-10T06:07:24.872 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T06:07:24.872 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T06:07:24.878 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T06:07:24.878 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-02-10T06:07:24.885 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout enabled 2024-02-10T06:07:24.891 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout active 2024-02-10T06:07:24.891 INFO:teuthology.orchestra.run.smithi089.stdout:Unit chronyd.service is enabled and running 2024-02-10T06:07:24.892 INFO:teuthology.orchestra.run.smithi089.stdout:Repeating the final host check... 2024-02-10T06:07:24.914 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T06:07:24.915 INFO:teuthology.orchestra.run.smithi089.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-10T06:07:24.915 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl is present 2024-02-10T06:07:24.915 INFO:teuthology.orchestra.run.smithi089.stdout:lvcreate is present 2024-02-10T06:07:24.921 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T06:07:24.921 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T06:07:24.927 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T06:07:24.928 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-02-10T06:07:24.935 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout enabled 2024-02-10T06:07:24.941 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout active 2024-02-10T06:07:24.941 INFO:teuthology.orchestra.run.smithi089.stdout:Unit chronyd.service is enabled and running 2024-02-10T06:07:24.941 INFO:teuthology.orchestra.run.smithi089.stdout:Host looks OK 2024-02-10T06:07:24.941 INFO:teuthology.orchestra.run.smithi089.stdout:Cluster fsid: 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:07:24.941 INFO:teuthology.orchestra.run.smithi089.stdout:Acquiring lock 139967659474608 on /run/cephadm/84e5b1ba-c7da-11ee-95b8-87774f69a715.lock 2024-02-10T06:07:24.941 INFO:teuthology.orchestra.run.smithi089.stdout:Lock 139967659474608 acquired on /run/cephadm/84e5b1ba-c7da-11ee-95b8-87774f69a715.lock 2024-02-10T06:07:24.941 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying IP 172.21.15.89 port 3300 ... 2024-02-10T06:07:24.942 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying IP 172.21.15.89 port 6789 ... 2024-02-10T06:07:24.942 INFO:teuthology.orchestra.run.smithi089.stdout:Base mon IP(s) is [172.21.15.89:3300, 172.21.15.89:6789], mon addrv is [v2:172.21.15.89:3300,v1:172.21.15.89:6789] 2024-02-10T06:07:24.945 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.89 metric 100 2024-02-10T06:07:24.945 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.89 metric 100 2024-02-10T06:07:24.948 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T06:07:24.948 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-10T06:07:24.948 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1772sec hoplimit 64 pref medium 2024-02-10T06:07:24.951 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T06:07:24.951 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-10T06:07:24.951 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T06:07:24.951 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-10T06:07:24.951 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-02-10T06:07:24.951 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T06:07:24.952 INFO:teuthology.orchestra.run.smithi089.stdout:Mon IP `172.21.15.89` is in CIDR network `172.21.0.0/20` 2024-02-10T06:07:24.952 INFO:teuthology.orchestra.run.smithi089.stdout:Mon IP `172.21.15.89` is in CIDR network `172.21.0.0/20` 2024-02-10T06:07:24.952 INFO:teuthology.orchestra.run.smithi089.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-10T06:07:24.952 INFO:teuthology.orchestra.run.smithi089.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T06:07:24.955 INFO:teuthology.orchestra.run.smithi089.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-10T06:07:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 2024-02-10T06:07:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-10T06:07:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Getting image source signatures 2024-02-10T06:07:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying blob sha256:a61c475ff6e77d421bceb2aff73188667ff70f4da71ee8d2a5b09b5c5d4ca045 2024-02-10T06:07:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-10T06:07:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying config sha256:a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 2024-02-10T06:07:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-10T06:07:27.103 INFO:teuthology.orchestra.run.smithi089.stdout:ceph: stdout ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable) 2024-02-10T06:07:27.103 INFO:teuthology.orchestra.run.smithi089.stdout:Ceph version: ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable) 2024-02-10T06:07:27.103 INFO:teuthology.orchestra.run.smithi089.stdout:Extracting ceph user uid/gid from container image... 2024-02-10T06:07:27.968 INFO:teuthology.orchestra.run.smithi089.stdout:stat: stdout 167 167 2024-02-10T06:07:27.968 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial keys... 2024-02-10T06:07:28.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQCgEsdlgFffERAAX+ZLoheH29VDYOvvGlDEKw== 2024-02-10T06:07:29.865 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQChEsdlQ7rbCRAA0fQ/gTcF1BkuDgLnEGhyLA== 2024-02-10T06:07:30.730 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQCiEsdlkUfBCRAA9r/BRk7NdEvhKYVE+7b9/A== 2024-02-10T06:07:30.732 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial monmap... 2024-02-10T06:07:35.206 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T06:07:35.206 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-02-10T06:07:35.206 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:07:35.206 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T06:07:35.206 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool for smithi089 [v2:172.21.15.89:3300,v1:172.21.15.89:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T06:07:35.206 INFO:teuthology.orchestra.run.smithi089.stdout:setting min_mon_release = octopus 2024-02-10T06:07:35.206 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: set fsid to 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:07:35.206 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T06:07:35.206 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:07:35.207 INFO:teuthology.orchestra.run.smithi089.stdout:Creating mon... 2024-02-10T06:07:39.841 INFO:teuthology.orchestra.run.smithi089.stdout:create mon.smithi089 on 2024-02-10T06:07:40.045 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-10T06:07:40.178 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T06:07:40.319 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715.target. 2024-02-10T06:07:40.320 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715.target. 2024-02-10T06:07:40.487 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mon.smithi089 2024-02-10T06:07:40.487 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to reset failed state of unit ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mon.smithi089.service: Unit ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mon.smithi089.service not loaded. 2024-02-10T06:07:40.649 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715.target.wants/ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mon.smithi089.service → /etc/systemd/system/ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@.service. 2024-02-10T06:07:41.158 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:41 smithi089 podman[28830]: 2024-02-10 06:07:41.094888638 +0000 UTC m=+0.323900937 container start dbc210811d23eb832e1a16a99e819fc0843531dd6ab867f0eafe15b178915aa6 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-02-10T06:07:41.222 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T06:07:41.222 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-02-10T06:07:41.229 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T06:07:41.229 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-02-10T06:07:41.229 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-02-10T06:07:41.229 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T06:07:41.229 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mon to start... 2024-02-10T06:07:41.229 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mon... 2024-02-10T06:07:41.450 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:41 smithi089 ceph-mon[28849]: mkfs 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:07:41.451 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:41 smithi089 ceph-mon[28849]: mon.smithi089 is new leader, mons smithi089 in quorum (ranks 0) 2024-02-10T06:07:41.451 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:41 smithi089 bash[28830]: dbc210811d23eb832e1a16a99e819fc0843531dd6ab867f0eafe15b178915aa6 2024-02-10T06:07:41.451 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:41 smithi089 systemd[1]: Started Ceph mon.smithi089 for 84e5b1ba-c7da-11ee-95b8-87774f69a715. 2024-02-10T06:07:42.299 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:42 smithi089 ceph-mon[28849]: mon.smithi089 is new leader, mons smithi089 in quorum (ranks 0) 2024-02-10T06:07:42.299 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:42 smithi089 ceph-mon[28849]: monmap e1: 1 mons at {smithi089=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:07:42.299 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:42 smithi089 ceph-mon[28849]: fsmap 2024-02-10T06:07:42.299 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:42 smithi089 ceph-mon[28849]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T06:07:42.299 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:42 smithi089 ceph-mon[28849]: mgrmap e1: no daemons active 2024-02-10T06:07:42.299 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:42 smithi089 ceph-mon[28849]: from='client.? 172.21.15.89:0/2595620937' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T06:07:42.307 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout cluster: 2024-02-10T06:07:42.307 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout id: 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:07:42.307 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-10T06:07:42.307 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:07:42.307 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout services: 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi089 (age 0.561647s) 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout data: 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout pgs: 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.640+0000 7f47be392700 1 Processor -- start 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.640+0000 7f47be392700 1 -- start start 2024-02-10T06:07:42.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.640+0000 7f47be392700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8104230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.640+0000 7f47be392700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b8104770 con 0x7f47b8103e20 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.640+0000 7f47b7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8104230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.640+0000 7f47b7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8104230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48026/0 (socket says 172.21.15.89:48026) 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.640+0000 7f47b7fff700 1 -- 172.21.15.89:0/3821524444 learned_addr learned my addr 172.21.15.89:0/3821524444 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.641+0000 7f47b7fff700 1 -- 172.21.15.89:0/3821524444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b81048b0 con 0x7f47b8103e20 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.641+0000 7f47b7fff700 1 --2- 172.21.15.89:0/3821524444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8104230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f47a80098e0 tx=0x7f47a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9844d717bdf9dcac server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.641+0000 7f47b6ffd700 1 -- 172.21.15.89:0/3821524444 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a800b860 con 0x7f47b8103e20 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.641+0000 7f47b6ffd700 1 -- 172.21.15.89:0/3821524444 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a800b9c0 con 0x7f47b8103e20 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.641+0000 7f47b6ffd700 1 -- 172.21.15.89:0/3821524444 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a800bc90 con 0x7f47b8103e20 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.642+0000 7f47be392700 1 -- 172.21.15.89:0/3821524444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 msgr2=0x7f47b8104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.642+0000 7f47be392700 1 --2- 172.21.15.89:0/3821524444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8104230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f47a80098e0 tx=0x7f47a8009250 comp rx=0 tx=0).stop 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.642+0000 7f47be392700 1 -- 172.21.15.89:0/3821524444 shutdown_connections 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.642+0000 7f47be392700 1 --2- 172.21.15.89:0/3821524444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8104230 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:42.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.642+0000 7f47be392700 1 -- 172.21.15.89:0/3821524444 >> 172.21.15.89:0/3821524444 conn(0x7f47b80fc580 msgr2=0x7f47b80fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.642+0000 7f47be392700 1 -- 172.21.15.89:0/3821524444 shutdown_connections 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.642+0000 7f47be392700 1 -- 172.21.15.89:0/3821524444 wait complete. 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.642+0000 7f47be392700 1 Processor -- start 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.642+0000 7f47be392700 1 -- start start 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.642+0000 7f47be392700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8111030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.642+0000 7f47be392700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b8111570 con 0x7f47b8103e20 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47b7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8111030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47b7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8111030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48038/0 (socket says 172.21.15.89:48038) 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47b7fff700 1 -- 172.21.15.89:0/2595620937 learned_addr learned my addr 172.21.15.89:0/2595620937 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47b7fff700 1 -- 172.21.15.89:0/2595620937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a80094f0 con 0x7f47b8103e20 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47b7fff700 1 --2- 172.21.15.89:0/2595620937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8111030 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f47a8003080 tx=0x7f47a8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47b4ff9700 1 -- 172.21.15.89:0/2595620937 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a801c5c0 con 0x7f47b8103e20 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47b4ff9700 1 -- 172.21.15.89:0/2595620937 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a801c720 con 0x7f47b8103e20 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47b4ff9700 1 -- 172.21.15.89:0/2595620937 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a801ca30 con 0x7f47b8103e20 2024-02-10T06:07:42.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47be392700 1 -- 172.21.15.89:0/2595620937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b8111770 con 0x7f47b8103e20 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47be392700 1 -- 172.21.15.89:0/2595620937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b8112410 con 0x7f47b8103e20 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47b4ff9700 1 -- 172.21.15.89:0/2595620937 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f47a801a450 con 0x7f47b8103e20 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.643+0000 7f47b4ff9700 1 -- 172.21.15.89:0/2595620937 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f47a802e790 con 0x7f47b8103e20 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.644+0000 7f47be392700 1 -- 172.21.15.89:0/2595620937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b805f5c0 con 0x7f47b8103e20 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.646+0000 7f47b4ff9700 1 -- 172.21.15.89:0/2595620937 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f47a801a890 con 0x7f47b8103e20 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.715+0000 7f47be392700 1 -- 172.21.15.89:0/2595620937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f47b8064ad0 con 0x7f47b8103e20 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.716+0000 7f47b4ff9700 1 -- 172.21.15.89:0/2595620937 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f47a801aa30 con 0x7f47b8103e20 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.716+0000 7f47be392700 1 -- 172.21.15.89:0/2595620937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 msgr2=0x7f47b8111030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.716+0000 7f47be392700 1 --2- 172.21.15.89:0/2595620937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8111030 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f47a8003080 tx=0x7f47a8016000 comp rx=0 tx=0).stop 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.716+0000 7f47be392700 1 -- 172.21.15.89:0/2595620937 shutdown_connections 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.716+0000 7f47be392700 1 --2- 172.21.15.89:0/2595620937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47b8103e20 0x7f47b8111030 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.717+0000 7f47be392700 1 -- 172.21.15.89:0/2595620937 >> 172.21.15.89:0/2595620937 conn(0x7f47b80fc580 msgr2=0x7f47b8107e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.717+0000 7f47be392700 1 -- 172.21.15.89:0/2595620937 shutdown_connections 2024-02-10T06:07:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:41.717+0000 7f47be392700 1 -- 172.21.15.89:0/2595620937 wait complete. 2024-02-10T06:07:42.312 INFO:teuthology.orchestra.run.smithi089.stdout:mon is available 2024-02-10T06:07:42.312 INFO:teuthology.orchestra.run.smithi089.stdout:Assimilating anything we can from ceph.conf... 2024-02-10T06:07:43.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:43 smithi089 ceph-mon[28849]: from='client.? 172.21.15.89:0/1618435439' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T06:07:43.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:43 smithi089 ceph-mon[28849]: from='client.? 172.21.15.89:0/1618435439' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T06:07:43.750 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:07:43.757 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T06:07:43.757 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout fsid = 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:07:43.757 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.89:3300,v1:172.21.15.89:6789] 2024-02-10T06:07:43.757 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T06:07:43.758 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.979+0000 7f4a411bd700 1 Processor -- start 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.979+0000 7f4a411bd700 1 -- start start 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.979+0000 7f4a411bd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.979+0000 7f4a411bd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c1030c0 con 0x7f4a3c102770 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.979+0000 7f4a3ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.980+0000 7f4a3ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48054/0 (socket says 172.21.15.89:48054) 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.980+0000 7f4a3ad9d700 1 -- 172.21.15.89:0/1768050862 learned_addr learned my addr 172.21.15.89:0/1768050862 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.980+0000 7f4a3ad9d700 1 -- 172.21.15.89:0/1768050862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c103200 con 0x7f4a3c102770 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.980+0000 7f4a3ad9d700 1 --2- 172.21.15.89:0/1768050862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c102b80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0098e0 tx=0x7f4a2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=106286862751c731 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.980+0000 7f4a39d9b700 1 -- 172.21.15.89:0/1768050862 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a2c00b860 con 0x7f4a3c102770 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.980+0000 7f4a39d9b700 1 -- 172.21.15.89:0/1768050862 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c00b9c0 con 0x7f4a3c102770 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.980+0000 7f4a39d9b700 1 -- 172.21.15.89:0/1768050862 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a2c00bc90 con 0x7f4a3c102770 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.980+0000 7f4a411bd700 1 -- 172.21.15.89:0/1768050862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 msgr2=0x7f4a3c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:43.759 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.980+0000 7f4a411bd700 1 --2- 172.21.15.89:0/1768050862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c102b80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0098e0 tx=0x7f4a2c009250 comp rx=0 tx=0).stop 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.980+0000 7f4a411bd700 1 -- 172.21.15.89:0/1768050862 shutdown_connections 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.981+0000 7f4a411bd700 1 --2- 172.21.15.89:0/1768050862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c102b80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.981+0000 7f4a411bd700 1 -- 172.21.15.89:0/1768050862 >> 172.21.15.89:0/1768050862 conn(0x7f4a3c0fded0 msgr2=0x7f4a3c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.981+0000 7f4a411bd700 1 -- 172.21.15.89:0/1768050862 shutdown_connections 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.981+0000 7f4a411bd700 1 -- 172.21.15.89:0/1768050862 wait complete. 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.981+0000 7f4a411bd700 1 Processor -- start 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.981+0000 7f4a411bd700 1 -- start start 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.981+0000 7f4a411bd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.981+0000 7f4a411bd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c077190 con 0x7f4a3c102770 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a3ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a3ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48058/0 (socket says 172.21.15.89:48058) 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a3ad9d700 1 -- 172.21.15.89:0/1618435439 learned_addr learned my addr 172.21.15.89:0/1618435439 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a3ad9d700 1 -- 172.21.15.89:0/1618435439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c0094f0 con 0x7f4a3c102770 2024-02-10T06:07:43.760 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a3ad9d700 1 --2- 172.21.15.89:0/1618435439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c078d10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c003080 tx=0x7f4a2c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a237fe700 1 -- 172.21.15.89:0/1618435439 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a2c01c5c0 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a237fe700 1 -- 172.21.15.89:0/1618435439 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c01c720 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a237fe700 1 -- 172.21.15.89:0/1618435439 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a2c01ca30 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a411bd700 1 -- 172.21.15.89:0/1618435439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3c077390 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a411bd700 1 -- 172.21.15.89:0/1618435439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3c077820 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a237fe700 1 -- 172.21.15.89:0/1618435439 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4a2c01a450 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.982+0000 7f4a237fe700 1 -- 172.21.15.89:0/1618435439 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4a2c02e790 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.983+0000 7f4a411bd700 1 -- 172.21.15.89:0/1618435439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3c10a640 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:42.985+0000 7f4a237fe700 1 -- 172.21.15.89:0/1618435439 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4a2c01a890 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:43.051+0000 7f4a411bd700 1 -- 172.21.15.89:0/1618435439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f4a3c103b10 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:43.055+0000 7f4a237fe700 1 -- 172.21.15.89:0/1618435439 <== mon.0 v2:172.21.15.89:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4a2c02ee80 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:43.055+0000 7f4a237fe700 1 -- 172.21.15.89:0/1618435439 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f4a2c024850 con 0x7f4a3c102770 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:43.056+0000 7f4a411bd700 1 -- 172.21.15.89:0/1618435439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 msgr2=0x7f4a3c078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:43.056+0000 7f4a411bd700 1 --2- 172.21.15.89:0/1618435439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c078d10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c003080 tx=0x7f4a2c016000 comp rx=0 tx=0).stop 2024-02-10T06:07:43.761 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:43.056+0000 7f4a411bd700 1 -- 172.21.15.89:0/1618435439 shutdown_connections 2024-02-10T06:07:43.762 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:43.056+0000 7f4a411bd700 1 --2- 172.21.15.89:0/1618435439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a3c102770 0x7f4a3c078d10 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:43.762 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:43.056+0000 7f4a411bd700 1 -- 172.21.15.89:0/1618435439 >> 172.21.15.89:0/1618435439 conn(0x7f4a3c0fded0 msgr2=0x7f4a3c1054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:43.762 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:43.056+0000 7f4a411bd700 1 -- 172.21.15.89:0/1618435439 shutdown_connections 2024-02-10T06:07:43.762 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:43.056+0000 7f4a411bd700 1 -- 172.21.15.89:0/1618435439 wait complete. 2024-02-10T06:07:43.762 INFO:teuthology.orchestra.run.smithi089.stdout:Generating new minimal ceph.conf... 2024-02-10T06:07:44.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:44 smithi089 ceph-mon[28849]: from='client.? 172.21.15.89:0/3277896709' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:07:46.294 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.188+0000 7f7f9377e700 1 Processor -- start 2024-02-10T06:07:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.188+0000 7f7f9377e700 1 -- start start 2024-02-10T06:07:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.188+0000 7f7f9377e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c1030c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.188+0000 7f7f9377e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c103600 con 0x7f7f8c102cb0 2024-02-10T06:07:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.188+0000 7f7f9151a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c1030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.188+0000 7f7f9151a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c1030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48070/0 (socket says 172.21.15.89:48070) 2024-02-10T06:07:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.188+0000 7f7f9151a700 1 -- 172.21.15.89:0/1739333266 learned_addr learned my addr 172.21.15.89:0/1739333266 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.189+0000 7f7f9151a700 1 -- 172.21.15.89:0/1739333266 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8c103740 con 0x7f7f8c102cb0 2024-02-10T06:07:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.189+0000 7f7f9151a700 1 --2- 172.21.15.89:0/1739333266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c1030c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7f880098e0 tx=0x7f7f88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f71a084bbfdddf2 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.189+0000 7f7f83fff700 1 -- 172.21.15.89:0/1739333266 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f8800b860 con 0x7f7f8c102cb0 2024-02-10T06:07:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.189+0000 7f7f83fff700 1 -- 172.21.15.89:0/1739333266 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7f8800be50 con 0x7f7f8c102cb0 2024-02-10T06:07:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.189+0000 7f7f83fff700 1 -- 172.21.15.89:0/1739333266 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f88010380 con 0x7f7f8c102cb0 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.189+0000 7f7f9377e700 1 -- 172.21.15.89:0/1739333266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 msgr2=0x7f7f8c1030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.189+0000 7f7f9377e700 1 --2- 172.21.15.89:0/1739333266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c1030c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7f880098e0 tx=0x7f7f88009250 comp rx=0 tx=0).stop 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.189+0000 7f7f9377e700 1 -- 172.21.15.89:0/1739333266 shutdown_connections 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.189+0000 7f7f9377e700 1 --2- 172.21.15.89:0/1739333266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c1030c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.189+0000 7f7f9377e700 1 -- 172.21.15.89:0/1739333266 >> 172.21.15.89:0/1739333266 conn(0x7f7f8c0fc580 msgr2=0x7f7f8c0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.190+0000 7f7f9377e700 1 -- 172.21.15.89:0/1739333266 shutdown_connections 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.190+0000 7f7f9377e700 1 -- 172.21.15.89:0/1739333266 wait complete. 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.190+0000 7f7f9377e700 1 Processor -- start 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.190+0000 7f7f9377e700 1 -- start start 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.190+0000 7f7f9377e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.190+0000 7f7f9377e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c113f80 con 0x7f7f8c102cb0 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.190+0000 7f7f9151a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:46.296 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.190+0000 7f7f9151a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48082/0 (socket says 172.21.15.89:48082) 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.190+0000 7f7f9151a700 1 -- 172.21.15.89:0/3277896709 learned_addr learned my addr 172.21.15.89:0/3277896709 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.191+0000 7f7f9151a700 1 -- 172.21.15.89:0/3277896709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f880094f0 con 0x7f7f8c102cb0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.191+0000 7f7f9151a700 1 --2- 172.21.15.89:0/3277896709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c113a40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7f88006d40 tx=0x7f7f88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.191+0000 7f7f81ffb700 1 -- 172.21.15.89:0/3277896709 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f88010760 con 0x7f7f8c102cb0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.191+0000 7f7f81ffb700 1 -- 172.21.15.89:0/3277896709 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7f88010d50 con 0x7f7f8c102cb0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.191+0000 7f7f9377e700 1 -- 172.21.15.89:0/3277896709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8c114180 con 0x7f7f8c102cb0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.191+0000 7f7f81ffb700 1 -- 172.21.15.89:0/3277896709 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f88024d20 con 0x7f7f8c102cb0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.191+0000 7f7f9377e700 1 -- 172.21.15.89:0/3277896709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8c114e20 con 0x7f7f8c102cb0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.191+0000 7f7f81ffb700 1 -- 172.21.15.89:0/3277896709 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f7f8802e6c0 con 0x7f7f8c102cb0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.191+0000 7f7f81ffb700 1 -- 172.21.15.89:0/3277896709 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7f8801a940 con 0x7f7f8c102cb0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.192+0000 7f7f9377e700 1 -- 172.21.15.89:0/3277896709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8c05f5c0 con 0x7f7f8c102cb0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.194+0000 7f7f81ffb700 1 -- 172.21.15.89:0/3277896709 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f7f88029070 con 0x7f7f8c102cb0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.260+0000 7f7f9377e700 1 -- 172.21.15.89:0/3277896709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f7f8c064ad0 con 0x7f7f8c102cb0 2024-02-10T06:07:46.297 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.261+0000 7f7f81ffb700 1 -- 172.21.15.89:0/3277896709 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f7f880108c0 con 0x7f7f8c102cb0 2024-02-10T06:07:46.298 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.261+0000 7f7f9377e700 1 -- 172.21.15.89:0/3277896709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 msgr2=0x7f7f8c113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:46.298 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.261+0000 7f7f9377e700 1 --2- 172.21.15.89:0/3277896709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c113a40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7f88006d40 tx=0x7f7f88009250 comp rx=0 tx=0).stop 2024-02-10T06:07:46.298 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.261+0000 7f7f9377e700 1 -- 172.21.15.89:0/3277896709 shutdown_connections 2024-02-10T06:07:46.298 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.261+0000 7f7f9377e700 1 --2- 172.21.15.89:0/3277896709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c102cb0 0x7f7f8c113a40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:46.298 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.261+0000 7f7f9377e700 1 -- 172.21.15.89:0/3277896709 >> 172.21.15.89:0/3277896709 conn(0x7f7f8c0fc580 msgr2=0x7f7f8c0fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:46.298 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.262+0000 7f7f9377e700 1 -- 172.21.15.89:0/3277896709 shutdown_connections 2024-02-10T06:07:46.298 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:44.262+0000 7f7f9377e700 1 -- 172.21.15.89:0/3277896709 wait complete. 2024-02-10T06:07:46.298 INFO:teuthology.orchestra.run.smithi089.stdout:Restarting the monitor... 2024-02-10T06:07:46.557 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:46 smithi089 systemd[1]: Stopping Ceph mon.smithi089 for 84e5b1ba-c7da-11ee-95b8-87774f69a715... 2024-02-10T06:07:46.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[28845]: 2024-02-10T06:07:46.556+0000 7feabcd1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi089 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:07:46.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[28845]: 2024-02-10T06:07:46.556+0000 7feabcd1e700 -1 mon.smithi089@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T06:07:46.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:46 smithi089 podman[29094]: 2024-02-10 06:07:46.770725777 +0000 UTC m=+0.287205253 container died dbc210811d23eb832e1a16a99e819fc0843531dd6ab867f0eafe15b178915aa6 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-10T06:07:47.382 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 podman[29094]: 2024-02-10 06:07:47.172881617 +0000 UTC m=+0.689361090 container cleanup dbc210811d23eb832e1a16a99e819fc0843531dd6ab867f0eafe15b178915aa6 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, RELEASE=quincy-72ce7c1, ceph=True, org.label-schema.vendor=CentOS) 2024-02-10T06:07:47.382 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 bash[29094]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089 2024-02-10T06:07:47.702 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 podman[29108]: 2024-02-10 06:07:47.38219925 +0000 UTC m=+0.615280104 container remove dbc210811d23eb832e1a16a99e819fc0843531dd6ab867f0eafe15b178915aa6 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:07:47.703 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 systemd[1]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mon.smithi089.service: Deactivated successfully. 2024-02-10T06:07:47.703 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 systemd[1]: Stopped Ceph mon.smithi089 for 84e5b1ba-c7da-11ee-95b8-87774f69a715. 2024-02-10T06:07:47.703 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 systemd[1]: Starting Ceph mon.smithi089 for 84e5b1ba-c7da-11ee-95b8-87774f69a715... 2024-02-10T06:07:47.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 podman[29199]: 2024-02-10 06:07:47.60560527 +0000 UTC m=+0.017095816 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph:quincy 2024-02-10T06:07:47.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 podman[29199]: 2024-02-10 06:07:47.724791438 +0000 UTC m=+0.136282019 container create 0109d6119759c20e843eb29825aa4cf2e87ada86c2823fffbf74f2e06a447a1a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=quincy-72ce7c1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-10T06:07:47.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 podman[29199]: 2024-02-10 06:07:47.877239512 +0000 UTC m=+0.288730051 container init 0109d6119759c20e843eb29825aa4cf2e87ada86c2823fffbf74f2e06a447a1a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:07:47.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 podman[29199]: 2024-02-10 06:07:47.879949172 +0000 UTC m=+0.291439712 container start 0109d6119759c20e843eb29825aa4cf2e87ada86c2823fffbf74f2e06a447a1a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:07:47.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T06:07:47.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable), process ceph-mon, pid 2 2024-02-10T06:07:47.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: pidfile_write: ignore empty --pid-file 2024-02-10T06:07:47.954 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: load: jerasure load: lrc 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: RocksDB version: 6.15.5 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Compile date Feb 9 2024 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: DB SUMMARY 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: DB Session ID: 4YYP34ZQCBGTQMM5Z6DN 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: CURRENT file: CURRENT 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi089/store.db dir, Total Num: 1, files: 000008.sst 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi089/store.db: 000010.log size: 97781 ; 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.error_if_exists: 0 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.create_if_missing: 0 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.paranoid_checks: 1 2024-02-10T06:07:47.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.env: 0x556ce6ebca80 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.fs: Posix File System 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.info_log: 0x556ce7d7fcc0 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.statistics: (nil) 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.use_fsync: 0 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_log_file_size: 0 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.allow_fallocate: 1 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T06:07:47.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.use_direct_reads: 0 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.db_log_dir: 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi089/store.db 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T06:07:47.957 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.write_buffer_manager: 0x556ce8af4b40 2024-02-10T06:07:47.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T06:07:47.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T06:07:47.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T06:07:47.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T06:07:47.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T06:07:47.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T06:07:47.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T06:07:47.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T06:07:47.958 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T06:07:47.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.unordered_write: 0 2024-02-10T06:07:47.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T06:07:47.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T06:07:47.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T06:07:47.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T06:07:47.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.row_cache: None 2024-02-10T06:07:47.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.wal_filter: None 2024-02-10T06:07:47.959 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.preserve_deletes: 0 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.two_write_queues: 0 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.atomic_flush: 0 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.log_readahead_size: 0 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T06:07:47.960 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_background_jobs: 2 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_background_compactions: -1 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_subcompactions: 1 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T06:07:47.961 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_open_files: -1 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_background_flushes: -1 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Compression algorithms supported: 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: kZSTD supported: 0 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: kXpressCompression supported: 0 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: kZlibCompression supported: 1 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: kSnappyCompression supported: 1 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: kLZ4Compression supported: 1 2024-02-10T06:07:47.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: kBZip2Compression supported: 0 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi089/store.db/MANIFEST-000009 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.merge_operator: 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_filter: None 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556ce7dd4f58) 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: cache_index_and_filter_blocks: 1 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T06:07:47.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: pin_top_level_index_and_filter: 1 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: index_type: 0 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: data_block_index_type: 0 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: index_shortening: 1 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: hash_index_allow_collision: 1 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: checksum: 1 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: no_block_cache: 0 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: block_cache: 0x556ce7e07090 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: block_cache_name: BinnedLRUCache 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: block_cache_options: 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: capacity : 536870912 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: num_shard_bits : 4 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: strict_capacity_limit : 0 2024-02-10T06:07:47.964 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: high_pri_pool_ratio: 0.000 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: block_cache_compressed: (nil) 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: persistent_cache: (nil) 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: block_size: 4096 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: block_size_deviation: 10 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: block_restart_interval: 16 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: index_block_restart_interval: 1 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: metadata_block_size: 4096 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: partition_filters: 0 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: use_delta_encoding: 1 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: whole_key_filtering: 1 2024-02-10T06:07:47.965 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: verify_compression: 0 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: read_amp_bytes_per_bit: 0 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: format_version: 4 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: enable_index_compression: 1 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: block_align: 0 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compression: NoCompression 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.num_levels: 7 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T06:07:47.966 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T06:07:47.967 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T06:07:47.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T06:07:47.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T06:07:47.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T06:07:47.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T06:07:47.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T06:07:47.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T06:07:47.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T06:07:47.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T06:07:47.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T06:07:47.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T06:07:47.968 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T06:07:47.969 INFO:teuthology.orchestra.run.smithi089.stdout:Setting mon public_network to 172.21.0.0/20 2024-02-10T06:07:47.969 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T06:07:47.970 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.table_properties_collectors: 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.inplace_update_support: 0 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T06:07:47.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.bloom_locality: 0 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.max_successive_merges: 0 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.ttl: 2592000 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.enable_blob_files: false 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.min_blob_size: 0 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T06:07:47.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi089/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707545267905686, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707545267906507, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92810, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90905, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13372, "raw_average_key_size": 49, "raw_value_size": 83606, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 270, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707545267, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "2e674b55-f543-47c8-8008-48ad98bf3286", "db_session_id": "4YYP34ZQCBGTQMM5Z6DN"}} 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707545267906815, "job": 1, "event": "recovery_finished"} 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi089/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x556ce7d9b180 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: DB pointer 0x556ce7e70000 2024-02-10T06:07:47.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: ** DB Stats ** 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: ** Compaction Stats [default] ** 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:07:47.974 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: L0 2/0 92.43 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.5 0.00 0.00 1 0.001 0 0 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Sum 2/0 92.43 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.5 0.00 0.00 1 0.001 0 0 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.5 0.00 0.00 1 0.001 0 0 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: ** Compaction Stats [default] ** 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.5 0.00 0.00 1 0.001 0 0 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Cumulative compaction: 0.00 GB write, 30.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:07:47.975 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Interval compaction: 0.00 GB write, 30.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: ** Compaction Stats [default] ** 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: L0 2/0 92.43 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.5 0.00 0.00 1 0.001 0 0 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Sum 2/0 92.43 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.5 0.00 0.00 1 0.001 0 0 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: ** Compaction Stats [default] ** 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:07:47.976 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.5 0.00 0.00 1 0.001 0 0 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Cumulative compaction: 0.00 GB write, 29.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: starting mon.smithi089 rank 0 at public addrs [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] at bind addrs [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi089 fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: mon.smithi089@-1(???) e1 preinit fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:07:47.977 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: mon.smithi089@-1(???).mds e1 new map 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: mon.smithi089@-1(???).mds e1 print_map 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: e1 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: legacy client fscid: -1 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout: No filesystems configured 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: mon.smithi089@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: mon.smithi089@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: mon.smithi089@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: mon.smithi089@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: mon.smithi089@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T06:07:47.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 bash[29199]: 0109d6119759c20e843eb29825aa4cf2e87ada86c2823fffbf74f2e06a447a1a 2024-02-10T06:07:47.979 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 systemd[1]: Started Ceph mon.smithi089 for 84e5b1ba-c7da-11ee-95b8-87774f69a715. 2024-02-10T06:07:48.305 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: mon.smithi089 is new leader, mons smithi089 in quorum (ranks 0) 2024-02-10T06:07:48.305 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: monmap e1: 1 mons at {smithi089=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:07:48.305 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: fsmap 2024-02-10T06:07:48.305 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T06:07:48.305 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:47 smithi089 ceph-mon[29217]: mgrmap e1: no daemons active 2024-02-10T06:07:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.406+0000 7f89c534c700 1 Processor -- start 2024-02-10T06:07:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.406+0000 7f89c534c700 1 -- start start 2024-02-10T06:07:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.406+0000 7f89c534c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c0102b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.406+0000 7f89c534c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c0103050 con 0x7f89c0102700 2024-02-10T06:07:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.406+0000 7f89beffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c0102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.406+0000 7f89beffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c0102b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48092/0 (socket says 172.21.15.89:48092) 2024-02-10T06:07:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.406+0000 7f89beffd700 1 -- 172.21.15.89:0/3731781947 learned_addr learned my addr 172.21.15.89:0/3731781947 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.407+0000 7f89beffd700 1 -- 172.21.15.89:0/3731781947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c0103190 con 0x7f89c0102700 2024-02-10T06:07:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.407+0000 7f89beffd700 1 --2- 172.21.15.89:0/3731781947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c0102b10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f89b00098e0 tx=0x7f89b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cefad2b02e77bf3 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.407+0000 7f89bdffb700 1 -- 172.21.15.89:0/3731781947 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b000b860 con 0x7f89c0102700 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.407+0000 7f89bdffb700 1 -- 172.21.15.89:0/3731781947 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f89b000be50 con 0x7f89c0102700 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.407+0000 7f89bdffb700 1 -- 172.21.15.89:0/3731781947 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b0010380 con 0x7f89c0102700 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.407+0000 7f89c534c700 1 -- 172.21.15.89:0/3731781947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 msgr2=0x7f89c0102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.407+0000 7f89c534c700 1 --2- 172.21.15.89:0/3731781947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c0102b10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f89b00098e0 tx=0x7f89b0009250 comp rx=0 tx=0).stop 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.407+0000 7f89c534c700 1 -- 172.21.15.89:0/3731781947 shutdown_connections 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.407+0000 7f89c534c700 1 --2- 172.21.15.89:0/3731781947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c0102b10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.407+0000 7f89c534c700 1 -- 172.21.15.89:0/3731781947 >> 172.21.15.89:0/3731781947 conn(0x7f89c00fde60 msgr2=0x7f89c0100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.408+0000 7f89c534c700 1 -- 172.21.15.89:0/3731781947 shutdown_connections 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.408+0000 7f89c534c700 1 -- 172.21.15.89:0/3731781947 wait complete. 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.408+0000 7f89c534c700 1 Processor -- start 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.408+0000 7f89c534c700 1 -- start start 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.408+0000 7f89c534c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c01114b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.408+0000 7f89c534c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c01119f0 con 0x7f89c0102700 2024-02-10T06:07:49.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.408+0000 7f89beffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c01114b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.408+0000 7f89beffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c01114b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48096/0 (socket says 172.21.15.89:48096) 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.408+0000 7f89beffd700 1 -- 172.21.15.89:0/3299152373 learned_addr learned my addr 172.21.15.89:0/3299152373 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.409+0000 7f89beffd700 1 -- 172.21.15.89:0/3299152373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b00094f0 con 0x7f89c0102700 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.409+0000 7f89beffd700 1 --2- 172.21.15.89:0/3299152373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c01114b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f89b000b440 tx=0x7f89b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.409+0000 7f89a77fe700 1 -- 172.21.15.89:0/3299152373 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b00107c0 con 0x7f89c0102700 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.409+0000 7f89a77fe700 1 -- 172.21.15.89:0/3299152373 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f89b0010db0 con 0x7f89c0102700 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.409+0000 7f89a77fe700 1 -- 172.21.15.89:0/3299152373 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b001ada0 con 0x7f89c0102700 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.409+0000 7f89c534c700 1 -- 172.21.15.89:0/3299152373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c0111bf0 con 0x7f89c0102700 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.409+0000 7f89c534c700 1 -- 172.21.15.89:0/3299152373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c0112890 con 0x7f89c0102700 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.409+0000 7f89a77fe700 1 -- 172.21.15.89:0/3299152373 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f89b002e6c0 con 0x7f89c0102700 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.409+0000 7f89a77fe700 1 -- 172.21.15.89:0/3299152373 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f89b0024950 con 0x7f89c0102700 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.410+0000 7f89c534c700 1 -- 172.21.15.89:0/3299152373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c005f5c0 con 0x7f89c0102700 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.412+0000 7f89a77fe700 1 -- 172.21.15.89:0/3299152373 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f89b0010920 con 0x7f89c0102700 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.476+0000 7f89c534c700 1 -- 172.21.15.89:0/3299152373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f89c0104570 con 0x7f89c0102700 2024-02-10T06:07:49.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.481+0000 7f89a77fe700 1 -- 172.21.15.89:0/3299152373 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f89b0010ac0 con 0x7f89c0102700 2024-02-10T06:07:49.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.482+0000 7f89c534c700 1 -- 172.21.15.89:0/3299152373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 msgr2=0x7f89c01114b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:49.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.482+0000 7f89c534c700 1 --2- 172.21.15.89:0/3299152373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c01114b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f89b000b440 tx=0x7f89b0009250 comp rx=0 tx=0).stop 2024-02-10T06:07:49.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.482+0000 7f89c534c700 1 -- 172.21.15.89:0/3299152373 shutdown_connections 2024-02-10T06:07:49.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.482+0000 7f89c534c700 1 --2- 172.21.15.89:0/3299152373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c0102700 0x7f89c01114b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:49.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.482+0000 7f89c534c700 1 -- 172.21.15.89:0/3299152373 >> 172.21.15.89:0/3299152373 conn(0x7f89c00fde60 msgr2=0x7f89c00ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:49.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.482+0000 7f89c534c700 1 -- 172.21.15.89:0/3299152373 shutdown_connections 2024-02-10T06:07:49.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:48.482+0000 7f89c534c700 1 -- 172.21.15.89:0/3299152373 wait complete. 2024-02-10T06:07:49.036 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-10T06:07:49.036 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:07:49.036 INFO:teuthology.orchestra.run.smithi089.stdout:Creating mgr... 2024-02-10T06:07:49.036 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying port 9283 ... 2024-02-10T06:07:49.203 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mgr.smithi089.pwmrye 2024-02-10T06:07:49.203 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to reset failed state of unit ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mgr.smithi089.pwmrye.service: Unit ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mgr.smithi089.pwmrye.service not loaded. 2024-02-10T06:07:49.347 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715.target.wants/ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mgr.smithi089.pwmrye.service → /etc/systemd/system/ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@.service. 2024-02-10T06:07:49.669 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:49 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3299152373' entity='client.admin' 2024-02-10T06:07:49.901 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T06:07:49.901 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-02-10T06:07:49.908 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T06:07:49.908 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-02-10T06:07:49.908 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-02-10T06:07:49.908 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T06:07:49.917 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T06:07:49.917 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-02-10T06:07:49.924 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T06:07:49.924 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-02-10T06:07:49.925 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-02-10T06:07:49.925 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-10T06:07:49.925 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr to start... 2024-02-10T06:07:49.925 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr... 2024-02-10T06:07:50.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:50 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/628552436' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T06:07:51.073 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:07:51.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:07:51.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "84e5b1ba-c7da-11ee-95b8-87774f69a715", 2024-02-10T06:07:51.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T06:07:51.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T06:07:51.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T06:07:51.076 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T06:07:51.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:51.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T06:07:51.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T06:07:51.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2024-02-10T06:07:51.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:07:51.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T06:07:51.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "smithi089" 2024-02-10T06:07:51.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:07:51.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-02-10T06:07:51.077 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T06:07:51.078 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:51.078 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-10T06:07:51.078 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T06:07:51.078 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:51.078 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T06:07:51.078 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:51.078 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T06:07:51.078 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T06:07:51.078 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T06:07:51.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T06:07:51.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T06:07:51.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T06:07:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T06:07:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T06:07:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T06:07:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T06:07:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T06:07:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T06:07:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T06:07:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T06:07:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T06:07:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:51.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T06:07:51.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:51.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T06:07:51.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T06:07:51.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:51.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T06:07:51.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T06:07:51.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T06:07:51.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T06:07:51.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T06:07:51.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T06:07:41.156491+0000", 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.396+0000 7fbdbc525700 1 Processor -- start 2024-02-10T06:07:51.090 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.396+0000 7fbdbc525700 1 -- start start 2024-02-10T06:07:51.091 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.396+0000 7fbdbc525700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:51.091 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.396+0000 7fbdbc525700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb4101fc0 con 0x7fbdb4101670 2024-02-10T06:07:51.091 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.396+0000 7fbdba2c1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:51.091 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.396+0000 7fbdba2c1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56520/0 (socket says 172.21.15.89:56520) 2024-02-10T06:07:51.091 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.396+0000 7fbdba2c1700 1 -- 172.21.15.89:0/3583947569 learned_addr learned my addr 172.21.15.89:0/3583947569 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:51.091 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.397+0000 7fbdba2c1700 1 -- 172.21.15.89:0/3583947569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb4102100 con 0x7fbdb4101670 2024-02-10T06:07:51.091 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.397+0000 7fbdba2c1700 1 --2- 172.21.15.89:0/3583947569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4101a80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00098e0 tx=0x7fbdb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265232ac23b15453 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:51.092 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.397+0000 7fbdb92bf700 1 -- 172.21.15.89:0/3583947569 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdb000b860 con 0x7fbdb4101670 2024-02-10T06:07:51.092 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.397+0000 7fbdb92bf700 1 -- 172.21.15.89:0/3583947569 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbdb000be50 con 0x7fbdb4101670 2024-02-10T06:07:51.092 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.397+0000 7fbdb92bf700 1 -- 172.21.15.89:0/3583947569 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdb0010380 con 0x7fbdb4101670 2024-02-10T06:07:51.092 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.398+0000 7fbdbc525700 1 -- 172.21.15.89:0/3583947569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 msgr2=0x7fbdb4101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:51.092 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.398+0000 7fbdbc525700 1 --2- 172.21.15.89:0/3583947569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4101a80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00098e0 tx=0x7fbdb0009250 comp rx=0 tx=0).stop 2024-02-10T06:07:51.092 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.398+0000 7fbdbc525700 1 -- 172.21.15.89:0/3583947569 shutdown_connections 2024-02-10T06:07:51.092 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.398+0000 7fbdbc525700 1 --2- 172.21.15.89:0/3583947569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4101a80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:51.092 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.398+0000 7fbdbc525700 1 -- 172.21.15.89:0/3583947569 >> 172.21.15.89:0/3583947569 conn(0x7fbdb40fcdd0 msgr2=0x7fbdb40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:51.092 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.398+0000 7fbdbc525700 1 -- 172.21.15.89:0/3583947569 shutdown_connections 2024-02-10T06:07:51.093 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.398+0000 7fbdbc525700 1 -- 172.21.15.89:0/3583947569 wait complete. 2024-02-10T06:07:51.093 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.399+0000 7fbdbc525700 1 Processor -- start 2024-02-10T06:07:51.093 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.399+0000 7fbdbc525700 1 -- start start 2024-02-10T06:07:51.093 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.399+0000 7fbdbc525700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4110470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:51.093 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.399+0000 7fbdbc525700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb41109b0 con 0x7fbdb4101670 2024-02-10T06:07:51.093 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.399+0000 7fbdba2c1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4110470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:51.093 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.399+0000 7fbdba2c1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4110470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56522/0 (socket says 172.21.15.89:56522) 2024-02-10T06:07:51.093 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.399+0000 7fbdba2c1700 1 -- 172.21.15.89:0/628552436 learned_addr learned my addr 172.21.15.89:0/628552436 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:51.093 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.399+0000 7fbdba2c1700 1 -- 172.21.15.89:0/628552436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb00094f0 con 0x7fbdb4101670 2024-02-10T06:07:51.093 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.399+0000 7fbdba2c1700 1 --2- 172.21.15.89:0/628552436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4110470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0000c00 tx=0x7fbdb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:51.093 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.400+0000 7fbdaaffd700 1 -- 172.21.15.89:0/628552436 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdb00107c0 con 0x7fbdb4101670 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.400+0000 7fbdaaffd700 1 -- 172.21.15.89:0/628552436 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbdb0010db0 con 0x7fbdb4101670 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.400+0000 7fbdaaffd700 1 -- 172.21.15.89:0/628552436 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdb001ad00 con 0x7fbdb4101670 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.400+0000 7fbdbc525700 1 -- 172.21.15.89:0/628552436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb4110bb0 con 0x7fbdb4101670 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.400+0000 7fbdbc525700 1 -- 172.21.15.89:0/628552436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb4111850 con 0x7fbdb4101670 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.400+0000 7fbdaaffd700 1 -- 172.21.15.89:0/628552436 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fbdb0024440 con 0x7fbdb4101670 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.400+0000 7fbdaaffd700 1 -- 172.21.15.89:0/628552436 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbdb0019710 con 0x7fbdb4101670 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.401+0000 7fbdbc525700 1 -- 172.21.15.89:0/628552436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb405f5c0 con 0x7fbdb4101670 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.403+0000 7fbdaaffd700 1 -- 172.21.15.89:0/628552436 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbdb00426d0 con 0x7fbdb4101670 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.474+0000 7fbdbc525700 1 -- 172.21.15.89:0/628552436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbdb41034e0 con 0x7fbdb4101670 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.475+0000 7fbdaaffd700 1 -- 172.21.15.89:0/628552436 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fbdb0031020 con 0x7fbdb4101670 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.475+0000 7fbdbc525700 1 -- 172.21.15.89:0/628552436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 msgr2=0x7fbdb4110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:51.094 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.475+0000 7fbdbc525700 1 --2- 172.21.15.89:0/628552436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4110470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0000c00 tx=0x7fbdb0009250 comp rx=0 tx=0).stop 2024-02-10T06:07:51.095 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.475+0000 7fbdbc525700 1 -- 172.21.15.89:0/628552436 shutdown_connections 2024-02-10T06:07:51.095 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.475+0000 7fbdbc525700 1 --2- 172.21.15.89:0/628552436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb4101670 0x7fbdb4110470 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:51.095 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.475+0000 7fbdbc525700 1 -- 172.21.15.89:0/628552436 >> 172.21.15.89:0/628552436 conn(0x7fbdb40fcdd0 msgr2=0x7fbdb40fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:51.095 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.475+0000 7fbdbc525700 1 -- 172.21.15.89:0/628552436 shutdown_connections 2024-02-10T06:07:51.095 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:50.475+0000 7fbdbc525700 1 -- 172.21.15.89:0/628552436 wait complete. 2024-02-10T06:07:51.095 INFO:teuthology.orchestra.run.smithi089.stdout:mgr not available, waiting (1/15)... 2024-02-10T06:07:54.194 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:53 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3022798977' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T06:07:54.217 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:07:54.218 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:07:54.218 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "84e5b1ba-c7da-11ee-95b8-87774f69a715", 2024-02-10T06:07:54.218 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T06:07:54.218 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T06:07:54.218 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T06:07:54.218 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T06:07:54.219 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:54.219 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T06:07:54.219 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T06:07:54.219 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2024-02-10T06:07:54.219 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:07:54.220 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T06:07:54.220 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "smithi089" 2024-02-10T06:07:54.220 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:07:54.220 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-02-10T06:07:54.220 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T06:07:54.220 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:54.221 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-10T06:07:54.221 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T06:07:54.221 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:54.221 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T06:07:54.221 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:54.221 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T06:07:54.222 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T06:07:54.222 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T06:07:54.222 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T06:07:54.222 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T06:07:54.222 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T06:07:54.222 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:54.223 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T06:07:54.223 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T06:07:54.223 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T06:07:54.223 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T06:07:54.223 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T06:07:54.223 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T06:07:54.224 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T06:07:54.224 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T06:07:54.224 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T06:07:54.224 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:54.224 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T06:07:54.224 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:54.224 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T06:07:54.224 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T06:07:54.225 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:54.225 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T06:07:54.225 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T06:07:54.225 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T06:07:54.225 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T06:07:54.225 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T06:07:54.225 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T06:07:54.225 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T06:07:54.225 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:07:54.226 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:07:54.226 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:54.226 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T06:07:54.226 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:54.226 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T06:07:41.156491+0000", 2024-02-10T06:07:54.226 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:07:54.226 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:54.226 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T06:07:54.226 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:07:54.227 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.537+0000 7ff5f15b3700 1 Processor -- start 2024-02-10T06:07:54.227 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.537+0000 7ff5f15b3700 1 -- start start 2024-02-10T06:07:54.227 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.537+0000 7ff5f15b3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec103070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:54.227 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.537+0000 7ff5f15b3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec1035b0 con 0x7ff5ec102c60 2024-02-10T06:07:54.227 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.538+0000 7ff5eaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:54.227 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.538+0000 7ff5eaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56526/0 (socket says 172.21.15.89:56526) 2024-02-10T06:07:54.227 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.538+0000 7ff5eaffd700 1 -- 172.21.15.89:0/4195663537 learned_addr learned my addr 172.21.15.89:0/4195663537 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:54.228 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.538+0000 7ff5eaffd700 1 -- 172.21.15.89:0/4195663537 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ec1036f0 con 0x7ff5ec102c60 2024-02-10T06:07:54.228 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.538+0000 7ff5eaffd700 1 --2- 172.21.15.89:0/4195663537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec103070 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0098e0 tx=0x7ff5dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4179afa0324765e server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:54.228 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.538+0000 7ff5e9ffb700 1 -- 172.21.15.89:0/4195663537 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5dc00b860 con 0x7ff5ec102c60 2024-02-10T06:07:54.228 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.538+0000 7ff5e9ffb700 1 -- 172.21.15.89:0/4195663537 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff5dc00be50 con 0x7ff5ec102c60 2024-02-10T06:07:54.228 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.538+0000 7ff5e9ffb700 1 -- 172.21.15.89:0/4195663537 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5dc010380 con 0x7ff5ec102c60 2024-02-10T06:07:54.228 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.539+0000 7ff5f15b3700 1 -- 172.21.15.89:0/4195663537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 msgr2=0x7ff5ec103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:54.228 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.539+0000 7ff5f15b3700 1 --2- 172.21.15.89:0/4195663537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec103070 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0098e0 tx=0x7ff5dc009250 comp rx=0 tx=0).stop 2024-02-10T06:07:54.228 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.539+0000 7ff5f15b3700 1 -- 172.21.15.89:0/4195663537 shutdown_connections 2024-02-10T06:07:54.228 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.539+0000 7ff5f15b3700 1 --2- 172.21.15.89:0/4195663537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec103070 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:54.228 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.539+0000 7ff5f15b3700 1 -- 172.21.15.89:0/4195663537 >> 172.21.15.89:0/4195663537 conn(0x7ff5ec0fc580 msgr2=0x7ff5ec0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.539+0000 7ff5f15b3700 1 -- 172.21.15.89:0/4195663537 shutdown_connections 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.539+0000 7ff5f15b3700 1 -- 172.21.15.89:0/4195663537 wait complete. 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.540+0000 7ff5f15b3700 1 Processor -- start 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.540+0000 7ff5f15b3700 1 -- start start 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.540+0000 7ff5f15b3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec116950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.540+0000 7ff5f15b3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec112340 con 0x7ff5ec102c60 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.540+0000 7ff5eaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec116950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.540+0000 7ff5eaffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec116950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56528/0 (socket says 172.21.15.89:56528) 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.540+0000 7ff5eaffd700 1 -- 172.21.15.89:0/3022798977 learned_addr learned my addr 172.21.15.89:0/3022798977 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.540+0000 7ff5eaffd700 1 -- 172.21.15.89:0/3022798977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc0094f0 con 0x7ff5ec102c60 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.540+0000 7ff5eaffd700 1 --2- 172.21.15.89:0/3022798977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec116950 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc006fd0 tx=0x7ff5dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.541+0000 7ff5d37fe700 1 -- 172.21.15.89:0/3022798977 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5dc0107c0 con 0x7ff5ec102c60 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.541+0000 7ff5f15b3700 1 -- 172.21.15.89:0/3022798977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ec101540 con 0x7ff5ec102c60 2024-02-10T06:07:54.229 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.541+0000 7ff5d37fe700 1 -- 172.21.15.89:0/3022798977 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff5dc010db0 con 0x7ff5ec102c60 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.541+0000 7ff5d37fe700 1 -- 172.21.15.89:0/3022798977 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5dc01add0 con 0x7ff5ec102c60 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.541+0000 7ff5f15b3700 1 -- 172.21.15.89:0/3022798977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ec116e90 con 0x7ff5ec102c60 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.541+0000 7ff5d37fe700 1 -- 172.21.15.89:0/3022798977 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff5dc022730 con 0x7ff5ec102c60 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.541+0000 7ff5d37fe700 1 -- 172.21.15.89:0/3022798977 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff5dc01e070 con 0x7ff5ec102c60 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.542+0000 7ff5f15b3700 1 -- 172.21.15.89:0/3022798977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ec10a570 con 0x7ff5ec102c60 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.545+0000 7ff5d37fe700 1 -- 172.21.15.89:0/3022798977 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ff5dc030060 con 0x7ff5ec102c60 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.616+0000 7ff5f15b3700 1 -- 172.21.15.89:0/3022798977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff5ec1025d0 con 0x7ff5ec102c60 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.617+0000 7ff5d37fe700 1 -- 172.21.15.89:0/3022798977 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7ff5dc01a5c0 con 0x7ff5ec102c60 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.618+0000 7ff5f15b3700 1 -- 172.21.15.89:0/3022798977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 msgr2=0x7ff5ec116950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.618+0000 7ff5f15b3700 1 --2- 172.21.15.89:0/3022798977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec116950 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc006fd0 tx=0x7ff5dc009250 comp rx=0 tx=0).stop 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.618+0000 7ff5f15b3700 1 -- 172.21.15.89:0/3022798977 shutdown_connections 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.618+0000 7ff5f15b3700 1 --2- 172.21.15.89:0/3022798977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5ec102c60 0x7ff5ec116950 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.618+0000 7ff5f15b3700 1 -- 172.21.15.89:0/3022798977 >> 172.21.15.89:0/3022798977 conn(0x7ff5ec0fc580 msgr2=0x7ff5ec0fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.618+0000 7ff5f15b3700 1 -- 172.21.15.89:0/3022798977 shutdown_connections 2024-02-10T06:07:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:53.618+0000 7ff5f15b3700 1 -- 172.21.15.89:0/3022798977 wait complete. 2024-02-10T06:07:54.231 INFO:teuthology.orchestra.run.smithi089.stdout:mgr not available, waiting (2/15)... 2024-02-10T06:07:56.716 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: Activating manager daemon smithi089.pwmrye 2024-02-10T06:07:56.716 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: mgrmap e2: smithi089.pwmrye(active, starting, since 0.00291718s) 2024-02-10T06:07:56.716 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: from='mgr.14100 172.21.15.89:0/480438845' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:07:56.716 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: from='mgr.14100 172.21.15.89:0/480438845' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:07:56.716 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: from='mgr.14100 172.21.15.89:0/480438845' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:07:56.716 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: from='mgr.14100 172.21.15.89:0/480438845' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-02-10T06:07:56.716 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: from='mgr.14100 172.21.15.89:0/480438845' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr metadata", "who": "smithi089.pwmrye", "id": "smithi089.pwmrye"}]: dispatch 2024-02-10T06:07:56.716 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: Manager daemon smithi089.pwmrye is now available 2024-02-10T06:07:56.716 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: from='mgr.14100 172.21.15.89:0/480438845' entity='mgr.smithi089.pwmrye' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.pwmrye/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:07:56.716 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: from='mgr.14100 172.21.15.89:0/480438845' entity='mgr.smithi089.pwmrye' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.pwmrye/trash_purge_schedule"}]: dispatch 2024-02-10T06:07:56.716 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: from='mgr.14100 172.21.15.89:0/480438845' entity='mgr.smithi089.pwmrye' 2024-02-10T06:07:56.717 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: from='mgr.14100 172.21.15.89:0/480438845' entity='mgr.smithi089.pwmrye' 2024-02-10T06:07:56.717 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:56 smithi089 ceph-mon[29217]: from='mgr.14100 172.21.15.89:0/480438845' entity='mgr.smithi089.pwmrye' 2024-02-10T06:07:57.236 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:07:57.246 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:07:57.246 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "84e5b1ba-c7da-11ee-95b8-87774f69a715", 2024-02-10T06:07:57.246 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T06:07:57.246 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "smithi089" 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-02-10T06:07:57.247 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:57.248 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:57.249 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T06:07:41.156491+0000", 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:07:57.250 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.639+0000 7f33a93e9700 1 Processor -- start 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.639+0000 7f33a93e9700 1 -- start start 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.639+0000 7f33a93e9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.639+0000 7f33a93e9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a40790b0 con 0x7f33a4078ca0 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.640+0000 7f33a2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.640+0000 7f33a2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56594/0 (socket says 172.21.15.89:56594) 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.640+0000 7f33a2ffd700 1 -- 172.21.15.89:0/4249725727 learned_addr learned my addr 172.21.15.89:0/4249725727 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.640+0000 7f33a2ffd700 1 -- 172.21.15.89:0/4249725727 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a40776f0 con 0x7f33a4078ca0 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.640+0000 7f33a2ffd700 1 --2- 172.21.15.89:0/4249725727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a40771b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3394009e50 tx=0x7f3394009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8993e5ae734b089b server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.640+0000 7f33a1ffb700 1 -- 172.21.15.89:0/4249725727 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33940092e0 con 0x7f33a4078ca0 2024-02-10T06:07:57.251 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.640+0000 7f33a1ffb700 1 -- 172.21.15.89:0/4249725727 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3394012430 con 0x7f33a4078ca0 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.640+0000 7f33a1ffb700 1 -- 172.21.15.89:0/4249725727 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33940075c0 con 0x7f33a4078ca0 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.641+0000 7f33a93e9700 1 -- 172.21.15.89:0/4249725727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 msgr2=0x7f33a40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.641+0000 7f33a93e9700 1 --2- 172.21.15.89:0/4249725727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a40771b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3394009e50 tx=0x7f3394009840 comp rx=0 tx=0).stop 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.641+0000 7f33a93e9700 1 -- 172.21.15.89:0/4249725727 shutdown_connections 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.641+0000 7f33a93e9700 1 --2- 172.21.15.89:0/4249725727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a40771b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.641+0000 7f33a93e9700 1 -- 172.21.15.89:0/4249725727 >> 172.21.15.89:0/4249725727 conn(0x7f33a40fded0 msgr2=0x7f33a4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.641+0000 7f33a93e9700 1 -- 172.21.15.89:0/4249725727 shutdown_connections 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.641+0000 7f33a93e9700 1 -- 172.21.15.89:0/4249725727 wait complete. 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f33a93e9700 1 Processor -- start 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f33a93e9700 1 -- start start 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f33a93e9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a41137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f33a93e9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a4113cf0 con 0x7f33a4078ca0 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f33a2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a41137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:07:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f33a2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a41137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56610/0 (socket says 172.21.15.89:56610) 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f33a2ffd700 1 -- 172.21.15.89:0/1015271919 learned_addr learned my addr 172.21.15.89:0/1015271919 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f33a2ffd700 1 -- 172.21.15.89:0/1015271919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3394009ae0 con 0x7f33a4078ca0 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f33a2ffd700 1 --2- 172.21.15.89:0/1015271919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a41137b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f33940127d0 tx=0x7f3394009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f339b7fe700 1 -- 172.21.15.89:0/1015271919 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33940092e0 con 0x7f33a4078ca0 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f339b7fe700 1 -- 172.21.15.89:0/1015271919 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3394007cc0 con 0x7f33a4078ca0 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f33a93e9700 1 -- 172.21.15.89:0/1015271919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a4113ef0 con 0x7f33a4078ca0 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.642+0000 7f339b7fe700 1 -- 172.21.15.89:0/1015271919 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f339401c970 con 0x7f33a4078ca0 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.643+0000 7f33a93e9700 1 -- 172.21.15.89:0/1015271919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a4114b90 con 0x7f33a4078ca0 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.643+0000 7f339b7fe700 1 -- 172.21.15.89:0/1015271919 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 2) v1 ==== 43124+0+0 (secure 0 0 0) 0x7f3394019070 con 0x7f33a4078ca0 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.643+0000 7f33a93e9700 1 -- 172.21.15.89:0/1015271919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a405f5c0 con 0x7f33a4078ca0 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.644+0000 7f339b7fe700 1 -- 172.21.15.89:0/1015271919 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3394049930 con 0x7f33a4078ca0 2024-02-10T06:07:57.253 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.646+0000 7f339b7fe700 1 -- 172.21.15.89:0/1015271919 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3394019b10 con 0x7f33a4078ca0 2024-02-10T06:07:57.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.714+0000 7f33a93e9700 1 -- 172.21.15.89:0/1015271919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f33a4114f10 con 0x7f33a4078ca0 2024-02-10T06:07:57.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.715+0000 7f339b7fe700 1 -- 172.21.15.89:0/1015271919 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f33940204f0 con 0x7f33a4078ca0 2024-02-10T06:07:57.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.716+0000 7f33a93e9700 1 -- 172.21.15.89:0/1015271919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 msgr2=0x7f33a41137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:07:57.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.716+0000 7f33a93e9700 1 --2- 172.21.15.89:0/1015271919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a41137b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f33940127d0 tx=0x7f3394009840 comp rx=0 tx=0).stop 2024-02-10T06:07:57.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.716+0000 7f33a93e9700 1 -- 172.21.15.89:0/1015271919 shutdown_connections 2024-02-10T06:07:57.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.716+0000 7f33a93e9700 1 --2- 172.21.15.89:0/1015271919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33a4078ca0 0x7f33a41137b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:07:57.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.716+0000 7f33a93e9700 1 -- 172.21.15.89:0/1015271919 >> 172.21.15.89:0/1015271919 conn(0x7f33a40fded0 msgr2=0x7f33a40ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:07:57.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.716+0000 7f33a93e9700 1 -- 172.21.15.89:0/1015271919 shutdown_connections 2024-02-10T06:07:57.254 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:56.716+0000 7f33a93e9700 1 -- 172.21.15.89:0/1015271919 wait complete. 2024-02-10T06:07:57.254 INFO:teuthology.orchestra.run.smithi089.stdout:mgr not available, waiting (3/15)... 2024-02-10T06:07:57.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:57 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1015271919' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T06:07:57.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:57 smithi089 ceph-mon[29217]: mgrmap e3: smithi089.pwmrye(active, since 1.00592s) 2024-02-10T06:08:00.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:59 smithi089 ceph-mon[29217]: mgrmap e4: smithi089.pwmrye(active, since 2s) 2024-02-10T06:08:00.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:07:59 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3246544776' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "84e5b1ba-c7da-11ee-95b8-87774f69a715", 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:08:00.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "smithi089" 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T06:08:00.454 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T06:08:00.455 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T06:08:00.456 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T06:07:41.156491+0000", 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.631+0000 7f779fa37700 1 Processor -- start 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.631+0000 7f779fa37700 1 -- start start 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.631+0000 7f779fa37700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.631+0000 7f779fa37700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7798101fc0 con 0x7f7798101670 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.631+0000 7f779d7d3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.631+0000 7f779d7d3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36378/0 (socket says 172.21.15.89:36378) 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.631+0000 7f779d7d3700 1 -- 172.21.15.89:0/1563199802 learned_addr learned my addr 172.21.15.89:0/1563199802 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.631+0000 7f779d7d3700 1 -- 172.21.15.89:0/1563199802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798102100 con 0x7f7798101670 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.632+0000 7f779d7d3700 1 --2- 172.21.15.89:0/1563199802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798101a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f779400bb80 tx=0x7f779400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722c72e9bfa8b300 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.632+0000 7f778ffff700 1 -- 172.21.15.89:0/1563199802 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f779400dd00 con 0x7f7798101670 2024-02-10T06:08:00.457 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.632+0000 7f778ffff700 1 -- 172.21.15.89:0/1563199802 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f779400de60 con 0x7f7798101670 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.632+0000 7f778ffff700 1 -- 172.21.15.89:0/1563199802 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77940188a0 con 0x7f7798101670 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.632+0000 7f779fa37700 1 -- 172.21.15.89:0/1563199802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 msgr2=0x7f7798101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.632+0000 7f779fa37700 1 --2- 172.21.15.89:0/1563199802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798101a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f779400bb80 tx=0x7f779400b4f0 comp rx=0 tx=0).stop 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.632+0000 7f779fa37700 1 -- 172.21.15.89:0/1563199802 shutdown_connections 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.632+0000 7f779fa37700 1 --2- 172.21.15.89:0/1563199802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798101a80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.632+0000 7f779fa37700 1 -- 172.21.15.89:0/1563199802 >> 172.21.15.89:0/1563199802 conn(0x7f77980fcdd0 msgr2=0x7f77980ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.633+0000 7f779fa37700 1 -- 172.21.15.89:0/1563199802 shutdown_connections 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.633+0000 7f779fa37700 1 -- 172.21.15.89:0/1563199802 wait complete. 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.633+0000 7f779fa37700 1 Processor -- start 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.633+0000 7f779fa37700 1 -- start start 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.633+0000 7f779fa37700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798113690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.633+0000 7f779fa37700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7798113bd0 con 0x7f7798101670 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.633+0000 7f779d7d3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798113690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.633+0000 7f779d7d3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798113690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36388/0 (socket says 172.21.15.89:36388) 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.633+0000 7f779d7d3700 1 -- 172.21.15.89:0/3246544776 learned_addr learned my addr 172.21.15.89:0/3246544776 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:00.458 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.634+0000 7f779d7d3700 1 -- 172.21.15.89:0/3246544776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779400b790 con 0x7f7798101670 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.634+0000 7f779d7d3700 1 --2- 172.21.15.89:0/3246544776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798113690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7794018460 tx=0x7f779400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.634+0000 7f778dffb700 1 -- 172.21.15.89:0/3246544776 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7794018e10 con 0x7f7798101670 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.634+0000 7f778dffb700 1 -- 172.21.15.89:0/3246544776 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7794010c00 con 0x7f7798101670 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.634+0000 7f779fa37700 1 -- 172.21.15.89:0/3246544776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7798113dd0 con 0x7f7798101670 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.634+0000 7f778dffb700 1 -- 172.21.15.89:0/3246544776 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7794021550 con 0x7f7798101670 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.634+0000 7f779fa37700 1 -- 172.21.15.89:0/3246544776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7798114a70 con 0x7f7798101670 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.635+0000 7f778dffb700 1 -- 172.21.15.89:0/3246544776 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f7794021780 con 0x7f7798101670 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.635+0000 7f778dffb700 1 --2- 172.21.15.89:0/3246544776 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f7784036790 0x7f7784038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.635+0000 7f779fa37700 1 -- 172.21.15.89:0/3246544776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779805f5c0 con 0x7f7798101670 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.635+0000 7f778dffb700 1 -- 172.21.15.89:0/3246544776 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7794014070 con 0x7f7798101670 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.635+0000 7f779cfd2700 1 --2- 172.21.15.89:0/3246544776 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f7784036790 0x7f7784038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.636+0000 7f779cfd2700 1 --2- 172.21.15.89:0/3246544776 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f7784036790 0x7f7784038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7788009990 tx=0x7f7788006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.640+0000 7f778dffb700 1 -- 172.21.15.89:0/3246544776 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7794021a30 con 0x7f7798101670 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.891+0000 7f779fa37700 1 -- 172.21.15.89:0/3246544776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f779804bed0 con 0x7f7798101670 2024-02-10T06:08:00.459 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.891+0000 7f778dffb700 1 -- 172.21.15.89:0/3246544776 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f779400e360 con 0x7f7798101670 2024-02-10T06:08:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.893+0000 7f779fa37700 1 -- 172.21.15.89:0/3246544776 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f7784036790 msgr2=0x7f7784038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.893+0000 7f779fa37700 1 --2- 172.21.15.89:0/3246544776 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f7784036790 0x7f7784038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7788009990 tx=0x7f7788006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.893+0000 7f779fa37700 1 -- 172.21.15.89:0/3246544776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 msgr2=0x7f7798113690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.893+0000 7f779fa37700 1 --2- 172.21.15.89:0/3246544776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798113690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7794018460 tx=0x7f779400b4f0 comp rx=0 tx=0).stop 2024-02-10T06:08:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.894+0000 7f779fa37700 1 -- 172.21.15.89:0/3246544776 shutdown_connections 2024-02-10T06:08:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.894+0000 7f779fa37700 1 --2- 172.21.15.89:0/3246544776 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f7784036790 0x7f7784038c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.894+0000 7f779fa37700 1 --2- 172.21.15.89:0/3246544776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7798101670 0x7f7798113690 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.894+0000 7f779fa37700 1 -- 172.21.15.89:0/3246544776 >> 172.21.15.89:0/3246544776 conn(0x7f77980fcdd0 msgr2=0x7f77980ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.894+0000 7f779fa37700 1 -- 172.21.15.89:0/3246544776 shutdown_connections 2024-02-10T06:08:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:07:59.894+0000 7f779fa37700 1 -- 172.21.15.89:0/3246544776 wait complete. 2024-02-10T06:08:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:mgr is available 2024-02-10T06:08:01.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:01 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2580272315' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T06:08:01.580 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:08:01.580 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout fsid = 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-10T06:08:01.581 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.828+0000 7fe829c63700 1 Processor -- start 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.829+0000 7fe829c63700 1 -- start start 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.829+0000 7fe829c63700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.829+0000 7fe829c63700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8241030c0 con 0x7fe824102770 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.829+0000 7fe8237fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.829+0000 7fe8237fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36394/0 (socket says 172.21.15.89:36394) 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.829+0000 7fe8237fe700 1 -- 172.21.15.89:0/2693474610 learned_addr learned my addr 172.21.15.89:0/2693474610 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.829+0000 7fe8237fe700 1 -- 172.21.15.89:0/2693474610 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe824103200 con 0x7fe824102770 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.829+0000 7fe8237fe700 1 --2- 172.21.15.89:0/2693474610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824102b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe8140098e0 tx=0x7fe814009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de22693751cd44e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.830+0000 7fe822ffd700 1 -- 172.21.15.89:0/2693474610 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe81400b860 con 0x7fe824102770 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.830+0000 7fe822ffd700 1 -- 172.21.15.89:0/2693474610 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe81400be50 con 0x7fe824102770 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.830+0000 7fe822ffd700 1 -- 172.21.15.89:0/2693474610 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe814010380 con 0x7fe824102770 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.830+0000 7fe829c63700 1 -- 172.21.15.89:0/2693474610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 msgr2=0x7fe824102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:01.582 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.830+0000 7fe829c63700 1 --2- 172.21.15.89:0/2693474610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824102b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe8140098e0 tx=0x7fe814009250 comp rx=0 tx=0).stop 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.830+0000 7fe829c63700 1 -- 172.21.15.89:0/2693474610 shutdown_connections 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.830+0000 7fe829c63700 1 --2- 172.21.15.89:0/2693474610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824102b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.830+0000 7fe829c63700 1 -- 172.21.15.89:0/2693474610 >> 172.21.15.89:0/2693474610 conn(0x7fe8240fded0 msgr2=0x7fe824100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.830+0000 7fe829c63700 1 -- 172.21.15.89:0/2693474610 shutdown_connections 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.830+0000 7fe829c63700 1 -- 172.21.15.89:0/2693474610 wait complete. 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.831+0000 7fe829c63700 1 Processor -- start 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.831+0000 7fe829c63700 1 -- start start 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.831+0000 7fe829c63700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.831+0000 7fe829c63700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe824111aa0 con 0x7fe824102770 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.831+0000 7fe8237fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.831+0000 7fe8237fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36398/0 (socket says 172.21.15.89:36398) 2024-02-10T06:08:01.583 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.831+0000 7fe8237fe700 1 -- 172.21.15.89:0/2580272315 learned_addr learned my addr 172.21.15.89:0/2580272315 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.831+0000 7fe8237fe700 1 -- 172.21.15.89:0/2580272315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8140094f0 con 0x7fe824102770 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.832+0000 7fe8237fe700 1 --2- 172.21.15.89:0/2580272315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824111560 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe81400b440 tx=0x7fe814009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.832+0000 7fe820ff9700 1 -- 172.21.15.89:0/2580272315 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe814010760 con 0x7fe824102770 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.832+0000 7fe820ff9700 1 -- 172.21.15.89:0/2580272315 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe814010d50 con 0x7fe824102770 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.832+0000 7fe829c63700 1 -- 172.21.15.89:0/2580272315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe824111ca0 con 0x7fe824102770 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.832+0000 7fe820ff9700 1 -- 172.21.15.89:0/2580272315 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe81401ae20 con 0x7fe824102770 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.832+0000 7fe829c63700 1 -- 172.21.15.89:0/2580272315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe824112940 con 0x7fe824102770 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.833+0000 7fe820ff9700 1 -- 172.21.15.89:0/2580272315 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fe81402e440 con 0x7fe824102770 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.833+0000 7fe829c63700 1 -- 172.21.15.89:0/2580272315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82405f5c0 con 0x7fe824102770 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.833+0000 7fe820ff9700 1 --2- 172.21.15.89:0/2580272315 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fe810036790 0x7fe810038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.833+0000 7fe820ff9700 1 -- 172.21.15.89:0/2580272315 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe81404a4e0 con 0x7fe824102770 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.833+0000 7fe81affd700 1 --2- 172.21.15.89:0/2580272315 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fe810036790 0x7fe810038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:01.584 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.833+0000 7fe81affd700 1 --2- 172.21.15.89:0/2580272315 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fe810036790 0x7fe810038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009990 tx=0x7fe80c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:00.837+0000 7fe820ff9700 1 -- 172.21.15.89:0/2580272315 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe814018940 con 0x7fe824102770 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.018+0000 7fe829c63700 1 -- 172.21.15.89:0/2580272315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe82404bed0 con 0x7fe824102770 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.019+0000 7fe820ff9700 1 -- 172.21.15.89:0/2580272315 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fe814018350 con 0x7fe824102770 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.021+0000 7fe829c63700 1 -- 172.21.15.89:0/2580272315 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fe810036790 msgr2=0x7fe810038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.021+0000 7fe829c63700 1 --2- 172.21.15.89:0/2580272315 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fe810036790 0x7fe810038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009990 tx=0x7fe80c006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.021+0000 7fe829c63700 1 -- 172.21.15.89:0/2580272315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 msgr2=0x7fe824111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.021+0000 7fe829c63700 1 --2- 172.21.15.89:0/2580272315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824111560 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe81400b440 tx=0x7fe814009250 comp rx=0 tx=0).stop 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.021+0000 7fe829c63700 1 -- 172.21.15.89:0/2580272315 shutdown_connections 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.021+0000 7fe829c63700 1 --2- 172.21.15.89:0/2580272315 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fe810036790 0x7fe810038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.021+0000 7fe829c63700 1 --2- 172.21.15.89:0/2580272315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe824102770 0x7fe824111560 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.021+0000 7fe829c63700 1 -- 172.21.15.89:0/2580272315 >> 172.21.15.89:0/2580272315 conn(0x7fe8240fded0 msgr2=0x7fe8240ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.022+0000 7fe829c63700 1 -- 172.21.15.89:0/2580272315 shutdown_connections 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.022+0000 7fe829c63700 1 -- 172.21.15.89:0/2580272315 wait complete. 2024-02-10T06:08:01.585 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling cephadm module... 2024-02-10T06:08:02.552 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:02 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1494597609' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T06:08:02.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.954+0000 7f322f4d6700 1 Processor -- start 2024-02-10T06:08:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.955+0000 7f322f4d6700 1 -- start start 2024-02-10T06:08:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.955+0000 7f322f4d6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f3228101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.955+0000 7f322f4d6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3228101fc0 con 0x7f3228101670 2024-02-10T06:08:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.955+0000 7f322d272700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f3228101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.955+0000 7f322d272700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f3228101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36404/0 (socket says 172.21.15.89:36404) 2024-02-10T06:08:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.955+0000 7f322d272700 1 -- 172.21.15.89:0/1814135107 learned_addr learned my addr 172.21.15.89:0/1814135107 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.955+0000 7f322d272700 1 -- 172.21.15.89:0/1814135107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3228102100 con 0x7f3228101670 2024-02-10T06:08:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.956+0000 7f322d272700 1 --2- 172.21.15.89:0/1814135107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f3228101a80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f32240098e0 tx=0x7f3224009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1f1b3009e9261a server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.956+0000 7f321ffff700 1 -- 172.21.15.89:0/1814135107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f322400b860 con 0x7f3228101670 2024-02-10T06:08:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.956+0000 7f321ffff700 1 -- 172.21.15.89:0/1814135107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f322400be50 con 0x7f3228101670 2024-02-10T06:08:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.956+0000 7f321ffff700 1 -- 172.21.15.89:0/1814135107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3224010380 con 0x7f3228101670 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.956+0000 7f322f4d6700 1 -- 172.21.15.89:0/1814135107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 msgr2=0x7f3228101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.956+0000 7f322f4d6700 1 --2- 172.21.15.89:0/1814135107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f3228101a80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f32240098e0 tx=0x7f3224009250 comp rx=0 tx=0).stop 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.956+0000 7f322f4d6700 1 -- 172.21.15.89:0/1814135107 shutdown_connections 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.956+0000 7f322f4d6700 1 --2- 172.21.15.89:0/1814135107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f3228101a80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.956+0000 7f322f4d6700 1 -- 172.21.15.89:0/1814135107 >> 172.21.15.89:0/1814135107 conn(0x7f32280fcdd0 msgr2=0x7f32280ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.956+0000 7f322f4d6700 1 -- 172.21.15.89:0/1814135107 shutdown_connections 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.956+0000 7f322f4d6700 1 -- 172.21.15.89:0/1814135107 wait complete. 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.957+0000 7f322f4d6700 1 Processor -- start 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.957+0000 7f322f4d6700 1 -- start start 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.957+0000 7f322f4d6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f32281077f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.957+0000 7f322f4d6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3228105c70 con 0x7f3228101670 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.957+0000 7f322d272700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f32281077f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.957+0000 7f322d272700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f32281077f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36416/0 (socket says 172.21.15.89:36416) 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.957+0000 7f322d272700 1 -- 172.21.15.89:0/1494597609 learned_addr learned my addr 172.21.15.89:0/1494597609 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.957+0000 7f322d272700 1 -- 172.21.15.89:0/1494597609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32240094f0 con 0x7f3228101670 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.958+0000 7f322d272700 1 --2- 172.21.15.89:0/1494597609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f32281077f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f322400b440 tx=0x7f3224009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.958+0000 7f321dffb700 1 -- 172.21.15.89:0/1494597609 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3224010760 con 0x7f3228101670 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.958+0000 7f321dffb700 1 -- 172.21.15.89:0/1494597609 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3224010d50 con 0x7f3228101670 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.958+0000 7f322f4d6700 1 -- 172.21.15.89:0/1494597609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3228105e70 con 0x7f3228101670 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.958+0000 7f321dffb700 1 -- 172.21.15.89:0/1494597609 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3224024d80 con 0x7f3228101670 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.958+0000 7f322f4d6700 1 -- 172.21.15.89:0/1494597609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3228106300 con 0x7f3228101670 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.959+0000 7f321dffb700 1 -- 172.21.15.89:0/1494597609 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f322401a430 con 0x7f3228101670 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.959+0000 7f322f4d6700 1 -- 172.21.15.89:0/1494597609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322810b4d0 con 0x7f3228101670 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.959+0000 7f321dffb700 1 --2- 172.21.15.89:0/1494597609 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f3214036790 0x7f3214038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:02.821 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.959+0000 7f321dffb700 1 -- 172.21.15.89:0/1494597609 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f322404a580 con 0x7f3228101670 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.959+0000 7f322ca71700 1 --2- 172.21.15.89:0/1494597609 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f3214036790 0x7f3214038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.959+0000 7f322ca71700 1 --2- 172.21.15.89:0/1494597609 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f3214036790 0x7f3214038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3218009990 tx=0x7f3218006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:01.963+0000 7f321dffb700 1 -- 172.21.15.89:0/1494597609 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f322401a6e0 con 0x7f3228101670 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.196+0000 7f322f4d6700 1 -- 172.21.15.89:0/1494597609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f3228064ad0 con 0x7f3228101670 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.249+0000 7f321dffb700 1 -- 172.21.15.89:0/1494597609 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f322402ad60 con 0x7f3228101670 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.250+0000 7f321dffb700 1 -- 172.21.15.89:0/1494597609 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f3224018530 con 0x7f3228101670 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.252+0000 7f322f4d6700 1 -- 172.21.15.89:0/1494597609 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f3214036790 msgr2=0x7f3214038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.252+0000 7f322f4d6700 1 --2- 172.21.15.89:0/1494597609 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f3214036790 0x7f3214038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3218009990 tx=0x7f3218006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.252+0000 7f322f4d6700 1 -- 172.21.15.89:0/1494597609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 msgr2=0x7f32281077f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.252+0000 7f322f4d6700 1 --2- 172.21.15.89:0/1494597609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f32281077f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f322400b440 tx=0x7f3224009250 comp rx=0 tx=0).stop 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.252+0000 7f322f4d6700 1 -- 172.21.15.89:0/1494597609 shutdown_connections 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.252+0000 7f322f4d6700 1 --2- 172.21.15.89:0/1494597609 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f3214036790 0x7f3214038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.252+0000 7f322f4d6700 1 --2- 172.21.15.89:0/1494597609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3228101670 0x7f32281077f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:02.823 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.252+0000 7f322f4d6700 1 -- 172.21.15.89:0/1494597609 >> 172.21.15.89:0/1494597609 conn(0x7f32280fcdd0 msgr2=0x7f32280fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:02.823 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.253+0000 7f322f4d6700 1 -- 172.21.15.89:0/1494597609 shutdown_connections 2024-02-10T06:08:02.823 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:02.253+0000 7f322f4d6700 1 -- 172.21.15.89:0/1494597609 wait complete. 2024-02-10T06:08:03.440 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:03 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1494597609' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T06:08:03.440 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:03 smithi089 ceph-mon[29217]: mgrmap e5: smithi089.pwmrye(active, since 5s) 2024-02-10T06:08:04.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:08:04.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "active_name": "smithi089.pwmrye", 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.235+0000 7f203ea28700 1 Processor -- start 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.235+0000 7f203ea28700 1 -- start start 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.235+0000 7f203ea28700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.235+0000 7f203ea28700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20381030c0 con 0x7f2038102770 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.235+0000 7f2037fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.235+0000 7f2037fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36436/0 (socket says 172.21.15.89:36436) 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.235+0000 7f2037fff700 1 -- 172.21.15.89:0/2200049907 learned_addr learned my addr 172.21.15.89:0/2200049907 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:04.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.235+0000 7f2037fff700 1 -- 172.21.15.89:0/2200049907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2038103200 con 0x7f2038102770 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.236+0000 7f2037fff700 1 --2- 172.21.15.89:0/2200049907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f20280098e0 tx=0x7f2028009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba50d31837655ae3 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.236+0000 7f2036ffd700 1 -- 172.21.15.89:0/2200049907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f202800b860 con 0x7f2038102770 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.236+0000 7f2036ffd700 1 -- 172.21.15.89:0/2200049907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f202800be50 con 0x7f2038102770 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.236+0000 7f2036ffd700 1 -- 172.21.15.89:0/2200049907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2028010380 con 0x7f2038102770 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.236+0000 7f203ea28700 1 -- 172.21.15.89:0/2200049907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 msgr2=0x7f2038102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.236+0000 7f203ea28700 1 --2- 172.21.15.89:0/2200049907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f20280098e0 tx=0x7f2028009250 comp rx=0 tx=0).stop 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.236+0000 7f203ea28700 1 -- 172.21.15.89:0/2200049907 shutdown_connections 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.236+0000 7f203ea28700 1 --2- 172.21.15.89:0/2200049907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038102b80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.236+0000 7f203ea28700 1 -- 172.21.15.89:0/2200049907 >> 172.21.15.89:0/2200049907 conn(0x7f20380fded0 msgr2=0x7f2038100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.236+0000 7f203ea28700 1 -- 172.21.15.89:0/2200049907 shutdown_connections 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.237+0000 7f203ea28700 1 -- 172.21.15.89:0/2200049907 wait complete. 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.237+0000 7f203ea28700 1 Processor -- start 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.237+0000 7f203ea28700 1 -- start start 2024-02-10T06:08:04.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.237+0000 7f203ea28700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.237+0000 7f203ea28700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2038077190 con 0x7f2038102770 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.237+0000 7f2037fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.237+0000 7f2037fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36446/0 (socket says 172.21.15.89:36446) 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.237+0000 7f2037fff700 1 -- 172.21.15.89:0/3343786495 learned_addr learned my addr 172.21.15.89:0/3343786495 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.238+0000 7f2037fff700 1 -- 172.21.15.89:0/3343786495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20280094f0 con 0x7f2038102770 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.238+0000 7f2037fff700 1 --2- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038078d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f202800b440 tx=0x7f2028009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.238+0000 7f2034ff9700 1 -- 172.21.15.89:0/3343786495 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20280107c0 con 0x7f2038102770 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.238+0000 7f2034ff9700 1 -- 172.21.15.89:0/3343786495 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2028010db0 con 0x7f2038102770 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.238+0000 7f203ea28700 1 -- 172.21.15.89:0/3343786495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2038077390 con 0x7f2038102770 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.238+0000 7f2034ff9700 1 -- 172.21.15.89:0/3343786495 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f202801ada0 con 0x7f2038102770 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.238+0000 7f203ea28700 1 -- 172.21.15.89:0/3343786495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2038077820 con 0x7f2038102770 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.239+0000 7f2034ff9700 1 -- 172.21.15.89:0/3343786495 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f2028024480 con 0x7f2038102770 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.239+0000 7f203ea28700 1 -- 172.21.15.89:0/3343786495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203804bed0 con 0x7f2038102770 2024-02-10T06:08:04.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.239+0000 7f2034ff9700 1 --2- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f2020036790 0x7f2020038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.239+0000 7f2034ff9700 1 -- 172.21.15.89:0/3343786495 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f202804b330 con 0x7f2038102770 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.239+0000 7f20377fe700 1 -- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f2020036790 msgr2=0x7f2020038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3711070832 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.239+0000 7f20377fe700 1 --2- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f2020036790 0x7f2020038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.244+0000 7f2034ff9700 1 -- 172.21.15.89:0/3343786495 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f202802e440 con 0x7f2038102770 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.440+0000 7f20377fe700 1 -- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f2020036790 msgr2=0x7f2020038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3711070832 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.440+0000 7f20377fe700 1 --2- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f2020036790 0x7f2020038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.529+0000 7f203ea28700 1 -- 172.21.15.89:0/3343786495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2038077f30 con 0x7f2038102770 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.530+0000 7f2034ff9700 1 -- 172.21.15.89:0/3343786495 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f202802cb80 con 0x7f2038102770 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.532+0000 7f203ea28700 1 -- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f2020036790 msgr2=0x7f2020038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.532+0000 7f203ea28700 1 --2- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f2020036790 0x7f2020038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.532+0000 7f203ea28700 1 -- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 msgr2=0x7f2038078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.532+0000 7f203ea28700 1 --2- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038078d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f202800b440 tx=0x7f2028009250 comp rx=0 tx=0).stop 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.532+0000 7f203ea28700 1 -- 172.21.15.89:0/3343786495 shutdown_connections 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.532+0000 7f203ea28700 1 --2- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7f2020036790 0x7f2020038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:04.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.532+0000 7f203ea28700 1 --2- 172.21.15.89:0/3343786495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2038102770 0x7f2038078d10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:04.156 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.532+0000 7f203ea28700 1 -- 172.21.15.89:0/3343786495 >> 172.21.15.89:0/3343786495 conn(0x7f20380fded0 msgr2=0x7f20380fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:04.156 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.532+0000 7f203ea28700 1 -- 172.21.15.89:0/3343786495 shutdown_connections 2024-02-10T06:08:04.156 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:03.532+0000 7f203ea28700 1 -- 172.21.15.89:0/3343786495 wait complete. 2024-02-10T06:08:04.156 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for the mgr to restart... 2024-02-10T06:08:04.156 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr epoch 5... 2024-02-10T06:08:04.408 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:04 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3343786495' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T06:08:08.915 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: Active manager daemon smithi089.pwmrye restarted 2024-02-10T06:08:08.915 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: Activating manager daemon smithi089.pwmrye 2024-02-10T06:08:08.915 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: osdmap e2: 0 total, 0 up, 0 in 2024-02-10T06:08:08.915 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: mgrmap e6: smithi089.pwmrye(active, starting, since 0.00320113s) 2024-02-10T06:08:08.915 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-02-10T06:08:08.915 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr metadata", "who": "smithi089.pwmrye", "id": "smithi089.pwmrye"}]: dispatch 2024-02-10T06:08:08.915 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:08:08.915 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:08:08.915 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:08:08.915 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: Manager daemon smithi089.pwmrye is now available 2024-02-10T06:08:08.916 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:08.916 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:08.916 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:08.916 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:08 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:10.166 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:09 smithi089 ceph-mon[29217]: Found migration_current of "None". Setting to last migration. 2024-02-10T06:08:10.166 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:09 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:10.166 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:09 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.pwmrye/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:08:10.166 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:09 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.pwmrye/trash_purge_schedule"}]: dispatch 2024-02-10T06:08:10.166 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:09 smithi089 ceph-mon[29217]: mgrmap e7: smithi089.pwmrye(active, since 1.00557s) 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.610+0000 7fd3eb28a700 1 Processor -- start 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.610+0000 7fd3eb28a700 1 -- start start 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.610+0000 7fd3eb28a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.610+0000 7fd3eb28a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e4101fc0 con 0x7fd3e4101670 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.610+0000 7fd3e9026700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.610+0000 7fd3e9026700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36458/0 (socket says 172.21.15.89:36458) 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.610+0000 7fd3e9026700 1 -- 172.21.15.89:0/3079171314 learned_addr learned my addr 172.21.15.89:0/3079171314 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:10.394 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.611+0000 7fd3e9026700 1 -- 172.21.15.89:0/3079171314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e4102100 con 0x7fd3e4101670 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.611+0000 7fd3e9026700 1 --2- 172.21.15.89:0/3079171314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4101a80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd3e00098e0 tx=0x7fd3e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774ff16965a687a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.611+0000 7fd3dbfff700 1 -- 172.21.15.89:0/3079171314 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e000b860 con 0x7fd3e4101670 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.611+0000 7fd3dbfff700 1 -- 172.21.15.89:0/3079171314 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd3e000be50 con 0x7fd3e4101670 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.611+0000 7fd3dbfff700 1 -- 172.21.15.89:0/3079171314 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e0010380 con 0x7fd3e4101670 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.612+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3079171314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 msgr2=0x7fd3e4101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.612+0000 7fd3eb28a700 1 --2- 172.21.15.89:0/3079171314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4101a80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd3e00098e0 tx=0x7fd3e0009250 comp rx=0 tx=0).stop 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.612+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3079171314 shutdown_connections 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.612+0000 7fd3eb28a700 1 --2- 172.21.15.89:0/3079171314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4101a80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.612+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3079171314 >> 172.21.15.89:0/3079171314 conn(0x7fd3e40fcdd0 msgr2=0x7fd3e40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.612+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3079171314 shutdown_connections 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.612+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3079171314 wait complete. 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.612+0000 7fd3eb28a700 1 Processor -- start 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.612+0000 7fd3eb28a700 1 -- start start 2024-02-10T06:08:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3eb28a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4107a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3eb28a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e4105f10 con 0x7fd3e4101670 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3e9026700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4107a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3e9026700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4107a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36462/0 (socket says 172.21.15.89:36462) 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3e9026700 1 -- 172.21.15.89:0/3311114544 learned_addr learned my addr 172.21.15.89:0/3311114544 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3e9026700 1 -- 172.21.15.89:0/3311114544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e00094f0 con 0x7fd3e4101670 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3e9026700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4107a90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000b440 tx=0x7fd3e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e0010760 con 0x7fd3e4101670 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3311114544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e4106110 con 0x7fd3e4101670 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd3e0010d50 con 0x7fd3e4101670 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e001ae20 con 0x7fd3e4101670 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.613+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3311114544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e41065a0 con 0x7fd3e4101670 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.614+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fd3e002e440 con 0x7fd3e4101670 2024-02-10T06:08:10.396 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.614+0000 7fd3d9ffb700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 0x7fd3d0038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.614+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 --> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3d0039330 con 0x7fd3d0036770 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.614+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd3e004a620 con 0x7fd3e4101670 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.614+0000 7fd3e8825700 1 -- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 msgr2=0x7fd3d0038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3711070832 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.614+0000 7fd3e8825700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 0x7fd3d0038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.815+0000 7fd3e8825700 1 -- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 msgr2=0x7fd3d0038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3711070832 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:04.815+0000 7fd3e8825700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 0x7fd3d0038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:05.216+0000 7fd3e8825700 1 -- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 msgr2=0x7fd3d0038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3711070832 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:05.216+0000 7fd3e8825700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 0x7fd3d0038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:06.017+0000 7fd3e8825700 1 -- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 msgr2=0x7fd3d0038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3711070832 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:06.017+0000 7fd3e8825700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 0x7fd3d0038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:07.617+0000 7fd3e8825700 1 -- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 msgr2=0x7fd3d0038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/3711070832 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:07.617+0000 7fd3e8825700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 0x7fd3d0038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:08.869+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7fd3e002e740 con 0x7fd3e4101670 2024-02-10T06:08:10.397 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:08.869+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 msgr2=0x7fd3d0038c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:08.869+0000 7fd3d9ffb700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 0x7fd3d0038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.870+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7fd3e002eac0 con 0x7fd3e4101670 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.870+0000 7fd3d9ffb700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd3d0039ff0 0x7fd3d003c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.870+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3d0039330 con 0x7fd3d0039ff0 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.871+0000 7fd3e8825700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd3d0039ff0 0x7fd3d003c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.872+0000 7fd3e8825700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd3d0039ff0 0x7fd3d003c3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40039d0 tx=0x7fd3d4009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.872+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fd3d0039330 con 0x7fd3d0039ff0 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.878+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3311114544 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd3e405f5c0 con 0x7fd3d0039ff0 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.878+0000 7fd3d9ffb700 1 -- 172.21.15.89:0/3311114544 <== mgr.14120 v2:172.21.15.89:6800/1397975713 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd3e405f5c0 con 0x7fd3d0039ff0 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.879+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd3d0039ff0 msgr2=0x7fd3d003c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.879+0000 7fd3eb28a700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd3d0039ff0 0x7fd3d003c3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40039d0 tx=0x7fd3d4009350 comp rx=0 tx=0).stop 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.879+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 msgr2=0x7fd3e4107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.879+0000 7fd3eb28a700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4107a90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000b440 tx=0x7fd3e0009250 comp rx=0 tx=0).stop 2024-02-10T06:08:10.398 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.879+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3311114544 shutdown_connections 2024-02-10T06:08:10.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.879+0000 7fd3eb28a700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd3d0039ff0 0x7fd3d003c3d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:10.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.879+0000 7fd3eb28a700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3e4101670 0x7fd3e4107a90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:10.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.879+0000 7fd3eb28a700 1 --2- 172.21.15.89:0/3311114544 >> [v2:172.21.15.89:6800/3711070832,v1:172.21.15.89:6801/3711070832] conn(0x7fd3d0036770 0x7fd3d0038c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:10.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.879+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3311114544 >> 172.21.15.89:0/3311114544 conn(0x7fd3e40fcdd0 msgr2=0x7fd3e4076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:10.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.879+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3311114544 shutdown_connections 2024-02-10T06:08:10.399 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:09.879+0000 7fd3eb28a700 1 -- 172.21.15.89:0/3311114544 wait complete. 2024-02-10T06:08:10.399 INFO:teuthology.orchestra.run.smithi089.stdout:mgr epoch 5 is available 2024-02-10T06:08:10.399 INFO:teuthology.orchestra.run.smithi089.stdout:Setting orchestrator backend to cephadm... 2024-02-10T06:08:11.532 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:11 smithi089 ceph-mon[29217]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T06:08:11.532 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:11 smithi089 ceph-mon[29217]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T06:08:11.532 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:11 smithi089 ceph-mon[29217]: [10/Feb/2024:06:08:10] ENGINE Bus STARTING 2024-02-10T06:08:11.532 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:11 smithi089 ceph-mon[29217]: [10/Feb/2024:06:08:10] ENGINE Serving on https://172.21.15.89:7150 2024-02-10T06:08:11.532 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:11 smithi089 ceph-mon[29217]: [10/Feb/2024:06:08:10] ENGINE Bus STARTED 2024-02-10T06:08:11.532 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:11 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:11.533 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:11 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:11.533 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:11 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:11.533 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:11 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:11.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.764+0000 7f33eb15c700 1 Processor -- start 2024-02-10T06:08:11.540 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.764+0000 7f33eb15c700 1 -- start start 2024-02-10T06:08:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.764+0000 7f33eb15c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.764+0000 7f33eb15c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e40790b0 con 0x7f33e4078ca0 2024-02-10T06:08:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.764+0000 7f33e8ef8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.765+0000 7f33e8ef8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59002/0 (socket says 172.21.15.89:59002) 2024-02-10T06:08:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.765+0000 7f33e8ef8700 1 -- 172.21.15.89:0/3870049204 learned_addr learned my addr 172.21.15.89:0/3870049204 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.765+0000 7f33e8ef8700 1 -- 172.21.15.89:0/3870049204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e40776f0 con 0x7f33e4078ca0 2024-02-10T06:08:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.765+0000 7f33e8ef8700 1 --2- 172.21.15.89:0/3870049204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e40771b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f33d80098e0 tx=0x7f33d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d35c19b73aa807 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.765+0000 7f33e37fe700 1 -- 172.21.15.89:0/3870049204 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33d800b860 con 0x7f33e4078ca0 2024-02-10T06:08:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.765+0000 7f33e37fe700 1 -- 172.21.15.89:0/3870049204 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f33d800be50 con 0x7f33e4078ca0 2024-02-10T06:08:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.765+0000 7f33e37fe700 1 -- 172.21.15.89:0/3870049204 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33d8010380 con 0x7f33e4078ca0 2024-02-10T06:08:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.766+0000 7f33eb15c700 1 -- 172.21.15.89:0/3870049204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 msgr2=0x7f33e40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:11.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.766+0000 7f33eb15c700 1 --2- 172.21.15.89:0/3870049204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e40771b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f33d80098e0 tx=0x7f33d8009250 comp rx=0 tx=0).stop 2024-02-10T06:08:11.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.766+0000 7f33eb15c700 1 -- 172.21.15.89:0/3870049204 shutdown_connections 2024-02-10T06:08:11.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.766+0000 7f33eb15c700 1 --2- 172.21.15.89:0/3870049204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e40771b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:11.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.766+0000 7f33eb15c700 1 -- 172.21.15.89:0/3870049204 >> 172.21.15.89:0/3870049204 conn(0x7f33e40fded0 msgr2=0x7f33e4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:11.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.766+0000 7f33eb15c700 1 -- 172.21.15.89:0/3870049204 shutdown_connections 2024-02-10T06:08:11.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.766+0000 7f33eb15c700 1 -- 172.21.15.89:0/3870049204 wait complete. 2024-02-10T06:08:11.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.766+0000 7f33eb15c700 1 Processor -- start 2024-02-10T06:08:11.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.766+0000 7f33eb15c700 1 -- start start 2024-02-10T06:08:11.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33eb15c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e4113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:11.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33eb15c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e4113cc0 con 0x7f33e4078ca0 2024-02-10T06:08:11.542 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33e8ef8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e4113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:11.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33e8ef8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e4113780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59018/0 (socket says 172.21.15.89:59018) 2024-02-10T06:08:11.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33e8ef8700 1 -- 172.21.15.89:0/1453053122 learned_addr learned my addr 172.21.15.89:0/1453053122 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:11.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33e8ef8700 1 -- 172.21.15.89:0/1453053122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d80094f0 con 0x7f33e4078ca0 2024-02-10T06:08:11.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33e8ef8700 1 --2- 172.21.15.89:0/1453053122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e4113780 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f33d8006b00 tx=0x7f33d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:11.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33e17fa700 1 -- 172.21.15.89:0/1453053122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33d80107c0 con 0x7f33e4078ca0 2024-02-10T06:08:11.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33e17fa700 1 -- 172.21.15.89:0/1453053122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f33d8010db0 con 0x7f33e4078ca0 2024-02-10T06:08:11.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33eb15c700 1 -- 172.21.15.89:0/1453053122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e4113ec0 con 0x7f33e4078ca0 2024-02-10T06:08:11.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33e17fa700 1 -- 172.21.15.89:0/1453053122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33d801acf0 con 0x7f33e4078ca0 2024-02-10T06:08:11.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.767+0000 7f33eb15c700 1 -- 172.21.15.89:0/1453053122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e4114b60 con 0x7f33e4078ca0 2024-02-10T06:08:11.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.768+0000 7f33e17fa700 1 -- 172.21.15.89:0/1453053122 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f33d80193c0 con 0x7f33e4078ca0 2024-02-10T06:08:11.543 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.768+0000 7f33eb15c700 1 -- 172.21.15.89:0/1453053122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e405f5c0 con 0x7f33e4078ca0 2024-02-10T06:08:11.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.768+0000 7f33e17fa700 1 --2- 172.21.15.89:0/1453053122 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f33d40366b0 0x7f33d4038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:11.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.768+0000 7f33e17fa700 1 -- 172.21.15.89:0/1453053122 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f33d804a440 con 0x7f33e4078ca0 2024-02-10T06:08:11.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.769+0000 7f33e3fff700 1 --2- 172.21.15.89:0/1453053122 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f33d40366b0 0x7f33d4038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:11.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.769+0000 7f33e3fff700 1 --2- 172.21.15.89:0/1453053122 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f33d40366b0 0x7f33d4038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009990 tx=0x7f33d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:11.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.773+0000 7f33e17fa700 1 -- 172.21.15.89:0/1453053122 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f33d8022930 con 0x7f33e4078ca0 2024-02-10T06:08:11.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.968+0000 7f33eb15c700 1 -- 172.21.15.89:0/1453053122 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f33e4100240 con 0x7f33d40366b0 2024-02-10T06:08:11.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.974+0000 7f33e17fa700 1 -- 172.21.15.89:0/1453053122 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f33e4100240 con 0x7f33d40366b0 2024-02-10T06:08:11.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.976+0000 7f33eb15c700 1 -- 172.21.15.89:0/1453053122 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f33d40366b0 msgr2=0x7f33d4038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:11.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.976+0000 7f33eb15c700 1 --2- 172.21.15.89:0/1453053122 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f33d40366b0 0x7f33d4038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009990 tx=0x7f33d0006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:11.544 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.976+0000 7f33eb15c700 1 -- 172.21.15.89:0/1453053122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 msgr2=0x7f33e4113780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:11.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.976+0000 7f33eb15c700 1 --2- 172.21.15.89:0/1453053122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e4113780 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f33d8006b00 tx=0x7f33d8009250 comp rx=0 tx=0).stop 2024-02-10T06:08:11.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.976+0000 7f33eb15c700 1 -- 172.21.15.89:0/1453053122 shutdown_connections 2024-02-10T06:08:11.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.976+0000 7f33eb15c700 1 --2- 172.21.15.89:0/1453053122 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f33d40366b0 0x7f33d4038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:11.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.976+0000 7f33eb15c700 1 --2- 172.21.15.89:0/1453053122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33e4078ca0 0x7f33e4113780 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:11.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.976+0000 7f33eb15c700 1 -- 172.21.15.89:0/1453053122 >> 172.21.15.89:0/1453053122 conn(0x7f33e40fded0 msgr2=0x7f33e40ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:11.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.976+0000 7f33eb15c700 1 -- 172.21.15.89:0/1453053122 shutdown_connections 2024-02-10T06:08:11.545 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:10.976+0000 7f33eb15c700 1 -- 172.21.15.89:0/1453053122 wait complete. 2024-02-10T06:08:12.405 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:12 smithi089 ceph-mon[29217]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:12.405 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:12 smithi089 ceph-mon[29217]: mgrmap e8: smithi089.pwmrye(active, since 2s) 2024-02-10T06:08:12.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-10T06:08:12.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.912+0000 7fb8177c1700 1 Processor -- start 2024-02-10T06:08:12.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.912+0000 7fb8177c1700 1 -- start start 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.912+0000 7fb8177c1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810105360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.912+0000 7fb8177c1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8101058a0 con 0x7fb8100714a0 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.912+0000 7fb81555d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810105360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.912+0000 7fb81555d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810105360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59026/0 (socket says 172.21.15.89:59026) 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.912+0000 7fb81555d700 1 -- 172.21.15.89:0/1054212568 learned_addr learned my addr 172.21.15.89:0/1054212568 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.912+0000 7fb81555d700 1 -- 172.21.15.89:0/1054212568 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8101059e0 con 0x7fb8100714a0 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.913+0000 7fb81555d700 1 --2- 172.21.15.89:0/1054212568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810105360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0098e0 tx=0x7fb80c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a14551f11712d04 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.913+0000 7fb807fff700 1 -- 172.21.15.89:0/1054212568 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb80c00b860 con 0x7fb8100714a0 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.913+0000 7fb807fff700 1 -- 172.21.15.89:0/1054212568 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb80c00be50 con 0x7fb8100714a0 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.913+0000 7fb807fff700 1 -- 172.21.15.89:0/1054212568 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb80c010380 con 0x7fb8100714a0 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.913+0000 7fb8177c1700 1 -- 172.21.15.89:0/1054212568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 msgr2=0x7fb810105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.913+0000 7fb8177c1700 1 --2- 172.21.15.89:0/1054212568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810105360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0098e0 tx=0x7fb80c009250 comp rx=0 tx=0).stop 2024-02-10T06:08:12.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.913+0000 7fb8177c1700 1 -- 172.21.15.89:0/1054212568 shutdown_connections 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.913+0000 7fb8177c1700 1 --2- 172.21.15.89:0/1054212568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810105360 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.913+0000 7fb8177c1700 1 -- 172.21.15.89:0/1054212568 >> 172.21.15.89:0/1054212568 conn(0x7fb8100fcdd0 msgr2=0x7fb8100ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.914+0000 7fb8177c1700 1 -- 172.21.15.89:0/1054212568 shutdown_connections 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.914+0000 7fb8177c1700 1 -- 172.21.15.89:0/1054212568 wait complete. 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.914+0000 7fb8177c1700 1 Processor -- start 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.914+0000 7fb8177c1700 1 -- start start 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.914+0000 7fb8177c1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810111580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.914+0000 7fb8177c1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb810111ac0 con 0x7fb8100714a0 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.914+0000 7fb81555d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810111580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.914+0000 7fb81555d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810111580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59030/0 (socket says 172.21.15.89:59030) 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.914+0000 7fb81555d700 1 -- 172.21.15.89:0/3424125593 learned_addr learned my addr 172.21.15.89:0/3424125593 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.915+0000 7fb81555d700 1 -- 172.21.15.89:0/3424125593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80c0094f0 con 0x7fb8100714a0 2024-02-10T06:08:12.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.915+0000 7fb81555d700 1 --2- 172.21.15.89:0/3424125593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810111580 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb80c003050 tx=0x7fb80c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.915+0000 7fb805ffb700 1 -- 172.21.15.89:0/3424125593 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb80c010760 con 0x7fb8100714a0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.915+0000 7fb805ffb700 1 -- 172.21.15.89:0/3424125593 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb80c010d50 con 0x7fb8100714a0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.915+0000 7fb8177c1700 1 -- 172.21.15.89:0/3424125593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb810111cc0 con 0x7fb8100714a0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.915+0000 7fb805ffb700 1 -- 172.21.15.89:0/3424125593 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb80c01ae20 con 0x7fb8100714a0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.915+0000 7fb8177c1700 1 -- 172.21.15.89:0/3424125593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb810112960 con 0x7fb8100714a0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.916+0000 7fb805ffb700 1 -- 172.21.15.89:0/3424125593 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb80c02e440 con 0x7fb8100714a0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.916+0000 7fb805ffb700 1 --2- 172.21.15.89:0/3424125593 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb7fc0367e0 0x7fb7fc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.916+0000 7fb805ffb700 1 -- 172.21.15.89:0/3424125593 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb80c04a5a0 con 0x7fb8100714a0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.916+0000 7fb814d5c700 1 --2- 172.21.15.89:0/3424125593 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb7fc0367e0 0x7fb7fc038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.916+0000 7fb8177c1700 1 -- 172.21.15.89:0/3424125593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb810112150 con 0x7fb8100714a0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.917+0000 7fb814d5c700 1 --2- 172.21.15.89:0/3424125593 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb7fc0367e0 0x7fb7fc038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb800009990 tx=0x7fb800006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:11.921+0000 7fb805ffb700 1 -- 172.21.15.89:0/3424125593 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb80c018940 con 0x7fb8100714a0 2024-02-10T06:08:12.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.105+0000 7fb8177c1700 1 -- 172.21.15.89:0/3424125593 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fb810112ba0 con 0x7fb7fc0367e0 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.106+0000 7fb805ffb700 1 -- 172.21.15.89:0/3424125593 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fb810112ba0 con 0x7fb7fc0367e0 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.108+0000 7fb8177c1700 1 -- 172.21.15.89:0/3424125593 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb7fc0367e0 msgr2=0x7fb7fc038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.108+0000 7fb8177c1700 1 --2- 172.21.15.89:0/3424125593 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb7fc0367e0 0x7fb7fc038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb800009990 tx=0x7fb800006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.108+0000 7fb8177c1700 1 -- 172.21.15.89:0/3424125593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 msgr2=0x7fb810111580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.108+0000 7fb8177c1700 1 --2- 172.21.15.89:0/3424125593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810111580 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb80c003050 tx=0x7fb80c009250 comp rx=0 tx=0).stop 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.108+0000 7fb8177c1700 1 -- 172.21.15.89:0/3424125593 shutdown_connections 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.108+0000 7fb8177c1700 1 --2- 172.21.15.89:0/3424125593 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb7fc0367e0 0x7fb7fc038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.108+0000 7fb8177c1700 1 --2- 172.21.15.89:0/3424125593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8100714a0 0x7fb810111580 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.108+0000 7fb8177c1700 1 -- 172.21.15.89:0/3424125593 >> 172.21.15.89:0/3424125593 conn(0x7fb8100fcdd0 msgr2=0x7fb8100fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.108+0000 7fb8177c1700 1 -- 172.21.15.89:0/3424125593 shutdown_connections 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:12.108+0000 7fb8177c1700 1 -- 172.21.15.89:0/3424125593 wait complete. 2024-02-10T06:08:12.683 INFO:teuthology.orchestra.run.smithi089.stdout:Generating ssh key... 2024-02-10T06:08:13.279 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:13 smithi089 ceph-mon[29217]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:13.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.086+0000 7f6917c6b700 1 Processor -- start 2024-02-10T06:08:13.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.087+0000 7f6917c6b700 1 -- start start 2024-02-10T06:08:13.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.087+0000 7f6917c6b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f69101041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:13.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.087+0000 7f6917c6b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69101047a0 con 0x7f6910101df0 2024-02-10T06:08:13.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.087+0000 7f6915a07700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f69101041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:13.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.087+0000 7f6915a07700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f69101041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59040/0 (socket says 172.21.15.89:59040) 2024-02-10T06:08:13.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.087+0000 7f6915a07700 1 -- 172.21.15.89:0/1401637676 learned_addr learned my addr 172.21.15.89:0/1401637676 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:13.894 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.087+0000 7f6915a07700 1 -- 172.21.15.89:0/1401637676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69101048e0 con 0x7f6910101df0 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.088+0000 7f6915a07700 1 --2- 172.21.15.89:0/1401637676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f69101041d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69040098e0 tx=0x7f6904009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb50f363f1f64a6b server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.088+0000 7f6914a05700 1 -- 172.21.15.89:0/1401637676 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f690400b860 con 0x7f6910101df0 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.088+0000 7f6914a05700 1 -- 172.21.15.89:0/1401637676 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f690400be50 con 0x7f6910101df0 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.088+0000 7f6914a05700 1 -- 172.21.15.89:0/1401637676 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6904010380 con 0x7f6910101df0 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.088+0000 7f6917c6b700 1 -- 172.21.15.89:0/1401637676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 msgr2=0x7f69101041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.088+0000 7f6917c6b700 1 --2- 172.21.15.89:0/1401637676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f69101041d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69040098e0 tx=0x7f6904009250 comp rx=0 tx=0).stop 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.088+0000 7f6917c6b700 1 -- 172.21.15.89:0/1401637676 shutdown_connections 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.088+0000 7f6917c6b700 1 --2- 172.21.15.89:0/1401637676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f69101041d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.088+0000 7f6917c6b700 1 -- 172.21.15.89:0/1401637676 >> 172.21.15.89:0/1401637676 conn(0x7f69100fc100 msgr2=0x7f69100fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.089+0000 7f6917c6b700 1 -- 172.21.15.89:0/1401637676 shutdown_connections 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.089+0000 7f6917c6b700 1 -- 172.21.15.89:0/1401637676 wait complete. 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.089+0000 7f6917c6b700 1 Processor -- start 2024-02-10T06:08:13.895 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.089+0000 7f6917c6b700 1 -- start start 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.089+0000 7f6917c6b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f6910111530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.089+0000 7f6917c6b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6910111a70 con 0x7f6910101df0 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.089+0000 7f6915a07700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f6910111530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.089+0000 7f6915a07700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f6910111530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59052/0 (socket says 172.21.15.89:59052) 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.089+0000 7f6915a07700 1 -- 172.21.15.89:0/3070469212 learned_addr learned my addr 172.21.15.89:0/3070469212 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.090+0000 7f6915a07700 1 -- 172.21.15.89:0/3070469212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69040094f0 con 0x7f6910101df0 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.090+0000 7f6915a07700 1 --2- 172.21.15.89:0/3070469212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f6910111530 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6904006b00 tx=0x7f6904009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.090+0000 7f69027fc700 1 -- 172.21.15.89:0/3070469212 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69040107c0 con 0x7f6910101df0 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.090+0000 7f69027fc700 1 -- 172.21.15.89:0/3070469212 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6904010db0 con 0x7f6910101df0 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.090+0000 7f6917c6b700 1 -- 172.21.15.89:0/3070469212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6910111cd0 con 0x7f6910101df0 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.090+0000 7f69027fc700 1 -- 172.21.15.89:0/3070469212 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f690401acf0 con 0x7f6910101df0 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.090+0000 7f6917c6b700 1 -- 172.21.15.89:0/3070469212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6910112910 con 0x7f6910101df0 2024-02-10T06:08:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.091+0000 7f69027fc700 1 -- 172.21.15.89:0/3070469212 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f69040193c0 con 0x7f6910101df0 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.091+0000 7f6917c6b700 1 -- 172.21.15.89:0/3070469212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691005f5c0 con 0x7f6910101df0 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.091+0000 7f69027fc700 1 --2- 172.21.15.89:0/3070469212 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f68fc0367e0 0x7f68fc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.091+0000 7f69027fc700 1 -- 172.21.15.89:0/3070469212 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f690404a6d0 con 0x7f6910101df0 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.091+0000 7f6915206700 1 --2- 172.21.15.89:0/3070469212 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f68fc0367e0 0x7f68fc038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.092+0000 7f6915206700 1 --2- 172.21.15.89:0/3070469212 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f68fc0367e0 0x7f68fc038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f690c009990 tx=0x7f690c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.096+0000 7f69027fc700 1 -- 172.21.15.89:0/3070469212 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6904022930 con 0x7f6910101df0 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.279+0000 7f6917c6b700 1 -- 172.21.15.89:0/3070469212 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f69100fe2e0 con 0x7f68fc0367e0 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.365+0000 7f69027fc700 1 -- 172.21.15.89:0/3070469212 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f69100fe2e0 con 0x7f68fc0367e0 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.367+0000 7f6917c6b700 1 -- 172.21.15.89:0/3070469212 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f68fc0367e0 msgr2=0x7f68fc038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.367+0000 7f6917c6b700 1 --2- 172.21.15.89:0/3070469212 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f68fc0367e0 0x7f68fc038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f690c009990 tx=0x7f690c006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.367+0000 7f6917c6b700 1 -- 172.21.15.89:0/3070469212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 msgr2=0x7f6910111530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.367+0000 7f6917c6b700 1 --2- 172.21.15.89:0/3070469212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f6910111530 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6904006b00 tx=0x7f6904009250 comp rx=0 tx=0).stop 2024-02-10T06:08:13.897 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.367+0000 7f6917c6b700 1 -- 172.21.15.89:0/3070469212 shutdown_connections 2024-02-10T06:08:13.898 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.367+0000 7f6917c6b700 1 --2- 172.21.15.89:0/3070469212 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f68fc0367e0 0x7f68fc038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:13.898 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.367+0000 7f6917c6b700 1 --2- 172.21.15.89:0/3070469212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6910101df0 0x7f6910111530 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:13.898 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.367+0000 7f6917c6b700 1 -- 172.21.15.89:0/3070469212 >> 172.21.15.89:0/3070469212 conn(0x7f69100fc100 msgr2=0x7f69101015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:13.898 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.367+0000 7f6917c6b700 1 -- 172.21.15.89:0/3070469212 shutdown_connections 2024-02-10T06:08:13.898 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:13.367+0000 7f6917c6b700 1 -- 172.21.15.89:0/3070469212 wait complete. 2024-02-10T06:08:14.294 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:14 smithi089 ceph-mon[29217]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:14.295 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:14 smithi089 ceph-mon[29217]: Generating ssh key... 2024-02-10T06:08:14.295 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:14 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:14.295 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:14 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:15.043 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiI1xJj862iCJ/a5TXfqajVgGO70gz8mGFoZtYuyyg51ze7eOXX7jmGDeP+I+O3klk+ORG8k2KLkfB1dWqlK6v4ZbCKGz9UC5A2UtKMO+fXNEd7tULCJxUjriS9Vb+aPNS8oOvFOXYflI6ynR/AaX+vFCPn0NAxcdZh36SRCkH1C7eEtBkPhLF9rx3OUf8IiAY6tLDE+zJpqhR8xQVxfpM2FRLBkdXSSul4DN5PCPlm8NlMvfmImBRSRmP1unXSq+EDjUDyvglawQ8gXM6VwbSC5AGWwYvl8NBfDlIUQEdCOQzsB4XlkRedabNibXtC/HLxGWebglTu8A/LPyxX6pg6iJfCV6U6nZuqoVsPZhTfr7iHyrLzhSW/tWJbaYlcC78G3KWJW4yrsg7pTryHzFe93aSwTvW4q/jBpDqm4nB1QoZF+E+o46qzetrRglHGXXk++ZPXxrD3+Z9z12Gs4HupfrVWmOPO0lwvJSeDmBPYLJxWpKHQdWXSz2mf3t4ztU= ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.294+0000 7fbb622c5700 1 Processor -- start 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.294+0000 7fbb622c5700 1 -- start start 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.294+0000 7fbb622c5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.294+0000 7fbb622c5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5c1030c0 con 0x7fbb5c102770 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.294+0000 7fbb5bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.294+0000 7fbb5bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59062/0 (socket says 172.21.15.89:59062) 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.294+0000 7fbb5bfff700 1 -- 172.21.15.89:0/1097580092 learned_addr learned my addr 172.21.15.89:0/1097580092 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.294+0000 7fbb5bfff700 1 -- 172.21.15.89:0/1097580092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c103200 con 0x7fbb5c102770 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.295+0000 7fbb5bfff700 1 --2- 172.21.15.89:0/1097580092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c102b80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c0098e0 tx=0x7fbb4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe99e265d783892 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.295+0000 7fbb5affd700 1 -- 172.21.15.89:0/1097580092 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb4c00b860 con 0x7fbb5c102770 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.295+0000 7fbb5affd700 1 -- 172.21.15.89:0/1097580092 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbb4c00be50 con 0x7fbb5c102770 2024-02-10T06:08:15.044 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.295+0000 7fbb5affd700 1 -- 172.21.15.89:0/1097580092 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb4c010380 con 0x7fbb5c102770 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.295+0000 7fbb622c5700 1 -- 172.21.15.89:0/1097580092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 msgr2=0x7fbb5c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.295+0000 7fbb622c5700 1 --2- 172.21.15.89:0/1097580092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c102b80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c0098e0 tx=0x7fbb4c009250 comp rx=0 tx=0).stop 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.295+0000 7fbb622c5700 1 -- 172.21.15.89:0/1097580092 shutdown_connections 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.295+0000 7fbb622c5700 1 --2- 172.21.15.89:0/1097580092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c102b80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.295+0000 7fbb622c5700 1 -- 172.21.15.89:0/1097580092 >> 172.21.15.89:0/1097580092 conn(0x7fbb5c0fded0 msgr2=0x7fbb5c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.295+0000 7fbb622c5700 1 -- 172.21.15.89:0/1097580092 shutdown_connections 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.295+0000 7fbb622c5700 1 -- 172.21.15.89:0/1097580092 wait complete. 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.296+0000 7fbb622c5700 1 Processor -- start 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.296+0000 7fbb622c5700 1 -- start start 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.296+0000 7fbb622c5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.296+0000 7fbb622c5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5c111aa0 con 0x7fbb5c102770 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.296+0000 7fbb5bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.296+0000 7fbb5bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59078/0 (socket says 172.21.15.89:59078) 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.296+0000 7fbb5bfff700 1 -- 172.21.15.89:0/1158145598 learned_addr learned my addr 172.21.15.89:0/1158145598 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:15.045 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.297+0000 7fbb5bfff700 1 -- 172.21.15.89:0/1158145598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c0094f0 con 0x7fbb5c102770 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.297+0000 7fbb5bfff700 1 --2- 172.21.15.89:0/1158145598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c111560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00b440 tx=0x7fbb4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.297+0000 7fbb58ff9700 1 -- 172.21.15.89:0/1158145598 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb4c010760 con 0x7fbb5c102770 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.297+0000 7fbb622c5700 1 -- 172.21.15.89:0/1158145598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5c111ca0 con 0x7fbb5c102770 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.297+0000 7fbb58ff9700 1 -- 172.21.15.89:0/1158145598 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbb4c010d50 con 0x7fbb5c102770 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.297+0000 7fbb58ff9700 1 -- 172.21.15.89:0/1158145598 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb4c01ae20 con 0x7fbb5c102770 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.297+0000 7fbb622c5700 1 -- 172.21.15.89:0/1158145598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5c112940 con 0x7fbb5c102770 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.298+0000 7fbb58ff9700 1 -- 172.21.15.89:0/1158145598 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fbb4c02e440 con 0x7fbb5c102770 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.298+0000 7fbb58ff9700 1 --2- 172.21.15.89:0/1158145598 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbb44036770 0x7fbb44038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.298+0000 7fbb58ff9700 1 -- 172.21.15.89:0/1158145598 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbb4c04b5b0 con 0x7fbb5c102770 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.298+0000 7fbb5b7fe700 1 --2- 172.21.15.89:0/1158145598 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbb44036770 0x7fbb44038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.298+0000 7fbb622c5700 1 -- 172.21.15.89:0/1158145598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5c112130 con 0x7fbb5c102770 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.298+0000 7fbb5b7fe700 1 --2- 172.21.15.89:0/1158145598 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbb44036770 0x7fbb44038c20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009990 tx=0x7fbb50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.303+0000 7fbb58ff9700 1 -- 172.21.15.89:0/1158145598 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbb4c018940 con 0x7fbb5c102770 2024-02-10T06:08:15.046 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.487+0000 7fbb622c5700 1 -- 172.21.15.89:0/1158145598 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb5c100190 con 0x7fbb44036770 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.488+0000 7fbb58ff9700 1 -- 172.21.15.89:0/1158145598 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fbb5c100190 con 0x7fbb44036770 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.490+0000 7fbb622c5700 1 -- 172.21.15.89:0/1158145598 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbb44036770 msgr2=0x7fbb44038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.490+0000 7fbb622c5700 1 --2- 172.21.15.89:0/1158145598 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbb44036770 0x7fbb44038c20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009990 tx=0x7fbb50006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.490+0000 7fbb622c5700 1 -- 172.21.15.89:0/1158145598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 msgr2=0x7fbb5c111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.490+0000 7fbb622c5700 1 --2- 172.21.15.89:0/1158145598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c111560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00b440 tx=0x7fbb4c009250 comp rx=0 tx=0).stop 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.490+0000 7fbb622c5700 1 -- 172.21.15.89:0/1158145598 shutdown_connections 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.490+0000 7fbb622c5700 1 --2- 172.21.15.89:0/1158145598 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbb44036770 0x7fbb44038c20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.490+0000 7fbb622c5700 1 --2- 172.21.15.89:0/1158145598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb5c102770 0x7fbb5c111560 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.490+0000 7fbb622c5700 1 -- 172.21.15.89:0/1158145598 >> 172.21.15.89:0/1158145598 conn(0x7fbb5c0fded0 msgr2=0x7fbb5c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.490+0000 7fbb622c5700 1 -- 172.21.15.89:0/1158145598 shutdown_connections 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:14.490+0000 7fbb622c5700 1 -- 172.21.15.89:0/1158145598 wait complete. 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:Adding key to root@localhost authorized_keys... 2024-02-10T06:08:15.047 INFO:teuthology.orchestra.run.smithi089.stdout:Adding host smithi089... 2024-02-10T06:08:15.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:15 smithi089 ceph-mon[29217]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:16.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:16 smithi089 ceph-mon[29217]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi089", "addr": "172.21.15.89", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:16.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:16 smithi089 ceph-mon[29217]: Deploying cephadm binary to smithi089 2024-02-10T06:08:18.185 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Added host 'smithi089' with addr '172.21.15.89' 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.437+0000 7f898fbef700 1 Processor -- start 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.437+0000 7f898fbef700 1 -- start start 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.437+0000 7f898fbef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.437+0000 7f898fbef700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8988101fc0 con 0x7f8988101670 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.437+0000 7f898d98b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.437+0000 7f898d98b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59082/0 (socket says 172.21.15.89:59082) 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.437+0000 7f898d98b700 1 -- 172.21.15.89:0/3966133046 learned_addr learned my addr 172.21.15.89:0/3966133046 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.437+0000 7f898d98b700 1 -- 172.21.15.89:0/3966133046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988102100 con 0x7f8988101670 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.438+0000 7f898d98b700 1 --2- 172.21.15.89:0/3966133046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988101a80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f89840098e0 tx=0x7f8984009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3223010767a5d965 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.438+0000 7f898c989700 1 -- 172.21.15.89:0/3966133046 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f898400b860 con 0x7f8988101670 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.438+0000 7f898c989700 1 -- 172.21.15.89:0/3966133046 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f898400be50 con 0x7f8988101670 2024-02-10T06:08:18.186 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.438+0000 7f898c989700 1 -- 172.21.15.89:0/3966133046 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8984010380 con 0x7f8988101670 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.438+0000 7f898fbef700 1 -- 172.21.15.89:0/3966133046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 msgr2=0x7f8988101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.438+0000 7f898fbef700 1 --2- 172.21.15.89:0/3966133046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988101a80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f89840098e0 tx=0x7f8984009250 comp rx=0 tx=0).stop 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.438+0000 7f898fbef700 1 -- 172.21.15.89:0/3966133046 shutdown_connections 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.438+0000 7f898fbef700 1 --2- 172.21.15.89:0/3966133046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988101a80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.438+0000 7f898fbef700 1 -- 172.21.15.89:0/3966133046 >> 172.21.15.89:0/3966133046 conn(0x7f89880fcdd0 msgr2=0x7f89880ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.438+0000 7f898fbef700 1 -- 172.21.15.89:0/3966133046 shutdown_connections 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.438+0000 7f898fbef700 1 -- 172.21.15.89:0/3966133046 wait complete. 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.439+0000 7f898fbef700 1 Processor -- start 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.439+0000 7f898fbef700 1 -- start start 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.439+0000 7f898fbef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988110740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.439+0000 7f898fbef700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8988110c80 con 0x7f8988101670 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.439+0000 7f898d98b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988110740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:18.187 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.439+0000 7f898d98b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988110740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59084/0 (socket says 172.21.15.89:59084) 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.439+0000 7f898d98b700 1 -- 172.21.15.89:0/3821533054 learned_addr learned my addr 172.21.15.89:0/3821533054 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.439+0000 7f898d98b700 1 -- 172.21.15.89:0/3821533054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89840094f0 con 0x7f8988101670 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.440+0000 7f898d98b700 1 --2- 172.21.15.89:0/3821533054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988110740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f898400b440 tx=0x7f8984009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.440+0000 7f897e7fc700 1 -- 172.21.15.89:0/3821533054 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8984010760 con 0x7f8988101670 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.440+0000 7f897e7fc700 1 -- 172.21.15.89:0/3821533054 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8984010d50 con 0x7f8988101670 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.440+0000 7f898fbef700 1 -- 172.21.15.89:0/3821533054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8988110e80 con 0x7f8988101670 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.440+0000 7f897e7fc700 1 -- 172.21.15.89:0/3821533054 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8984024d80 con 0x7f8988101670 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.440+0000 7f898fbef700 1 -- 172.21.15.89:0/3821533054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8988111b20 con 0x7f8988101670 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.441+0000 7f897e7fc700 1 -- 172.21.15.89:0/3821533054 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f898401a430 con 0x7f8988101670 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.441+0000 7f897e7fc700 1 --2- 172.21.15.89:0/3821533054 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f89740367e0 0x7f8974038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.441+0000 7f898fbef700 1 -- 172.21.15.89:0/3821533054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898805f5c0 con 0x7f8988101670 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.441+0000 7f897e7fc700 1 -- 172.21.15.89:0/3821533054 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f898404a820 con 0x7f8988101670 2024-02-10T06:08:18.188 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.441+0000 7f898d18a700 1 --2- 172.21.15.89:0/3821533054 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f89740367e0 0x7f8974038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.441+0000 7f898d18a700 1 --2- 172.21.15.89:0/3821533054 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f89740367e0 0x7f8974038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8978009990 tx=0x7f8978006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.445+0000 7f897e7fc700 1 -- 172.21.15.89:0/3821533054 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8984018940 con 0x7f8988101670 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:15.630+0000 7f898fbef700 1 -- 172.21.15.89:0/3821533054 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi089", "addr": "172.21.15.89", "target": ["mon-mgr", ""]}) v1 -- 0x7f8988103650 con 0x7f89740367e0 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:17.656+0000 7f897e7fc700 1 -- 172.21.15.89:0/3821533054 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f8988103650 con 0x7f89740367e0 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:17.659+0000 7f898fbef700 1 -- 172.21.15.89:0/3821533054 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f89740367e0 msgr2=0x7f8974038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:17.659+0000 7f898fbef700 1 --2- 172.21.15.89:0/3821533054 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f89740367e0 0x7f8974038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8978009990 tx=0x7f8978006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:17.659+0000 7f898fbef700 1 -- 172.21.15.89:0/3821533054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 msgr2=0x7f8988110740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:17.659+0000 7f898fbef700 1 --2- 172.21.15.89:0/3821533054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988110740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f898400b440 tx=0x7f8984009250 comp rx=0 tx=0).stop 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:17.659+0000 7f898fbef700 1 -- 172.21.15.89:0/3821533054 shutdown_connections 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:17.659+0000 7f898fbef700 1 --2- 172.21.15.89:0/3821533054 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f89740367e0 0x7f8974038c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:17.659+0000 7f898fbef700 1 --2- 172.21.15.89:0/3821533054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8988101670 0x7f8988110740 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:17.659+0000 7f898fbef700 1 -- 172.21.15.89:0/3821533054 >> 172.21.15.89:0/3821533054 conn(0x7f89880fcdd0 msgr2=0x7f89880fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:17.659+0000 7f898fbef700 1 -- 172.21.15.89:0/3821533054 shutdown_connections 2024-02-10T06:08:18.189 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:17.659+0000 7f898fbef700 1 -- 172.21.15.89:0/3821533054 wait complete. 2024-02-10T06:08:18.190 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying mon service with default placement... 2024-02-10T06:08:18.848 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:18 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:18.849 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:18 smithi089 ceph-mon[29217]: Added host smithi089 2024-02-10T06:08:18.849 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:18 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:19.803 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-10T06:08:19.803 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.610+0000 7f7ee9020700 1 Processor -- start 2024-02-10T06:08:19.803 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.610+0000 7f7ee9020700 1 -- start start 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.610+0000 7f7ee9020700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41031a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.610+0000 7f7ee9020700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee41036e0 con 0x7f7ee4102d90 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.610+0000 7f7ee2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.610+0000 7f7ee2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41666/0 (socket says 172.21.15.89:41666) 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.610+0000 7f7ee2d9d700 1 -- 172.21.15.89:0/414608250 learned_addr learned my addr 172.21.15.89:0/414608250 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.610+0000 7f7ee2d9d700 1 -- 172.21.15.89:0/414608250 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee4103820 con 0x7f7ee4102d90 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.611+0000 7f7ee2d9d700 1 --2- 172.21.15.89:0/414608250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41031a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40098e0 tx=0x7f7ed4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809042b07bd5a64a server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.611+0000 7f7ee1d9b700 1 -- 172.21.15.89:0/414608250 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ed400b860 con 0x7f7ee4102d90 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.611+0000 7f7ee1d9b700 1 -- 172.21.15.89:0/414608250 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7ed400be50 con 0x7f7ee4102d90 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.611+0000 7f7ee1d9b700 1 -- 172.21.15.89:0/414608250 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ed4010380 con 0x7f7ee4102d90 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.611+0000 7f7ee9020700 1 -- 172.21.15.89:0/414608250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 msgr2=0x7f7ee41031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.611+0000 7f7ee9020700 1 --2- 172.21.15.89:0/414608250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41031a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40098e0 tx=0x7f7ed4009250 comp rx=0 tx=0).stop 2024-02-10T06:08:19.804 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.611+0000 7f7ee9020700 1 -- 172.21.15.89:0/414608250 shutdown_connections 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.611+0000 7f7ee9020700 1 --2- 172.21.15.89:0/414608250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41031a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.611+0000 7f7ee9020700 1 -- 172.21.15.89:0/414608250 >> 172.21.15.89:0/414608250 conn(0x7f7ee40fc580 msgr2=0x7f7ee40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.611+0000 7f7ee9020700 1 -- 172.21.15.89:0/414608250 shutdown_connections 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.611+0000 7f7ee9020700 1 -- 172.21.15.89:0/414608250 wait complete. 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.612+0000 7f7ee9020700 1 Processor -- start 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.612+0000 7f7ee9020700 1 -- start start 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.612+0000 7f7ee9020700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.612+0000 7f7ee9020700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee4113f30 con 0x7f7ee4102d90 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.612+0000 7f7ee2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.612+0000 7f7ee2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41674/0 (socket says 172.21.15.89:41674) 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.612+0000 7f7ee2d9d700 1 -- 172.21.15.89:0/3324180731 learned_addr learned my addr 172.21.15.89:0/3324180731 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.613+0000 7f7ee2d9d700 1 -- 172.21.15.89:0/3324180731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed40094f0 con 0x7f7ee4102d90 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.613+0000 7f7ee2d9d700 1 --2- 172.21.15.89:0/3324180731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41139f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400b440 tx=0x7f7ed4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:19.805 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.613+0000 7f7ecb7fe700 1 -- 172.21.15.89:0/3324180731 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ed40107c0 con 0x7f7ee4102d90 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.613+0000 7f7ee9020700 1 -- 172.21.15.89:0/3324180731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee4114130 con 0x7f7ee4102d90 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.613+0000 7f7ecb7fe700 1 -- 172.21.15.89:0/3324180731 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7ed4010db0 con 0x7f7ee4102d90 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.613+0000 7f7ecb7fe700 1 -- 172.21.15.89:0/3324180731 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ed401ada0 con 0x7f7ee4102d90 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.613+0000 7f7ee9020700 1 -- 172.21.15.89:0/3324180731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee4114dd0 con 0x7f7ee4102d90 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.614+0000 7f7ecb7fe700 1 -- 172.21.15.89:0/3324180731 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7ed402e440 con 0x7f7ee4102d90 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.614+0000 7f7ee9020700 1 -- 172.21.15.89:0/3324180731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee405f5c0 con 0x7f7ee4102d90 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.614+0000 7f7ecb7fe700 1 --2- 172.21.15.89:0/3324180731 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f7ecc0367e0 0x7f7ecc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.614+0000 7f7ecb7fe700 1 -- 172.21.15.89:0/3324180731 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7ed404aca0 con 0x7f7ee4102d90 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.614+0000 7f7ee259c700 1 --2- 172.21.15.89:0/3324180731 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f7ecc0367e0 0x7f7ecc038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.614+0000 7f7ee259c700 1 --2- 172.21.15.89:0/3324180731 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f7ecc0367e0 0x7f7ecc038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009990 tx=0x7f7ed8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.619+0000 7f7ecb7fe700 1 -- 172.21.15.89:0/3324180731 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7ed4024440 con 0x7f7ee4102d90 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.847+0000 7f7ee9020700 1 -- 172.21.15.89:0/3324180731 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ee4108ba0 con 0x7f7ecc0367e0 2024-02-10T06:08:19.806 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.854+0000 7f7ecb7fe700 1 -- 172.21.15.89:0/3324180731 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7ee4108ba0 con 0x7f7ecc0367e0 2024-02-10T06:08:19.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.856+0000 7f7ee9020700 1 -- 172.21.15.89:0/3324180731 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f7ecc0367e0 msgr2=0x7f7ecc038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:19.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.856+0000 7f7ee9020700 1 --2- 172.21.15.89:0/3324180731 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f7ecc0367e0 0x7f7ecc038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009990 tx=0x7f7ed8006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:19.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.856+0000 7f7ee9020700 1 -- 172.21.15.89:0/3324180731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 msgr2=0x7f7ee41139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:19.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.856+0000 7f7ee9020700 1 --2- 172.21.15.89:0/3324180731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41139f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400b440 tx=0x7f7ed4009250 comp rx=0 tx=0).stop 2024-02-10T06:08:19.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.856+0000 7f7ee9020700 1 -- 172.21.15.89:0/3324180731 shutdown_connections 2024-02-10T06:08:19.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.856+0000 7f7ee9020700 1 --2- 172.21.15.89:0/3324180731 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f7ecc0367e0 0x7f7ecc038c90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:19.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.856+0000 7f7ee9020700 1 --2- 172.21.15.89:0/3324180731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ee4102d90 0x7f7ee41139f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:19.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.856+0000 7f7ee9020700 1 -- 172.21.15.89:0/3324180731 >> 172.21.15.89:0/3324180731 conn(0x7f7ee40fc580 msgr2=0x7f7ee40fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:19.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.857+0000 7f7ee9020700 1 -- 172.21.15.89:0/3324180731 shutdown_connections 2024-02-10T06:08:19.807 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:18.857+0000 7f7ee9020700 1 -- 172.21.15.89:0/3324180731 wait complete. 2024-02-10T06:08:19.807 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying mgr service with default placement... 2024-02-10T06:08:20.046 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:19 smithi089 ceph-mon[29217]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:20.046 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:19 smithi089 ceph-mon[29217]: Saving service mon spec with placement count:5 2024-02-10T06:08:20.046 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:19 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:21.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:21 smithi089 ceph-mon[29217]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:21 smithi089 ceph-mon[29217]: Saving service mgr spec with placement count:2 2024-02-10T06:08:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:21 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:21 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:21.883 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-10T06:08:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.227+0000 7f4611fad700 1 Processor -- start 2024-02-10T06:08:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.227+0000 7f4611fad700 1 -- start start 2024-02-10T06:08:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.227+0000 7f4611fad700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.227+0000 7f4611fad700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c1058c0 con 0x7f460c071480 2024-02-10T06:08:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.227+0000 7f460b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.227+0000 7f460b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41684/0 (socket says 172.21.15.89:41684) 2024-02-10T06:08:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.227+0000 7f460b7fe700 1 -- 172.21.15.89:0/2303802476 learned_addr learned my addr 172.21.15.89:0/2303802476 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.227+0000 7f460b7fe700 1 -- 172.21.15.89:0/2303802476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c105a00 con 0x7f460c071480 2024-02-10T06:08:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.227+0000 7f460b7fe700 1 --2- 172.21.15.89:0/2303802476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c105380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0098e0 tx=0x7f45fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4298fe338a83b3 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.227+0000 7f460affd700 1 -- 172.21.15.89:0/2303802476 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45fc00b860 con 0x7f460c071480 2024-02-10T06:08:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.227+0000 7f460affd700 1 -- 172.21.15.89:0/2303802476 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f45fc00be50 con 0x7f460c071480 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.228+0000 7f460affd700 1 -- 172.21.15.89:0/2303802476 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45fc010380 con 0x7f460c071480 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.228+0000 7f4611fad700 1 -- 172.21.15.89:0/2303802476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 msgr2=0x7f460c105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.228+0000 7f4611fad700 1 --2- 172.21.15.89:0/2303802476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c105380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0098e0 tx=0x7f45fc009250 comp rx=0 tx=0).stop 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.228+0000 7f4611fad700 1 -- 172.21.15.89:0/2303802476 shutdown_connections 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.228+0000 7f4611fad700 1 --2- 172.21.15.89:0/2303802476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c105380 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.228+0000 7f4611fad700 1 -- 172.21.15.89:0/2303802476 >> 172.21.15.89:0/2303802476 conn(0x7f460c0fcdd0 msgr2=0x7f460c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.228+0000 7f4611fad700 1 -- 172.21.15.89:0/2303802476 shutdown_connections 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.228+0000 7f4611fad700 1 -- 172.21.15.89:0/2303802476 wait complete. 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.229+0000 7f4611fad700 1 Processor -- start 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.229+0000 7f4611fad700 1 -- start start 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.229+0000 7f4611fad700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.229+0000 7f4611fad700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c101280 con 0x7f460c071480 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.229+0000 7f460b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.229+0000 7f460b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41694/0 (socket says 172.21.15.89:41694) 2024-02-10T06:08:21.885 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.229+0000 7f460b7fe700 1 -- 172.21.15.89:0/4047747564 learned_addr learned my addr 172.21.15.89:0/4047747564 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.229+0000 7f460b7fe700 1 -- 172.21.15.89:0/4047747564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc0094f0 con 0x7f460c071480 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.229+0000 7f460b7fe700 1 --2- 172.21.15.89:0/4047747564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c102e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f45fc003050 tx=0x7f45fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.230+0000 7f4608ff9700 1 -- 172.21.15.89:0/4047747564 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45fc010760 con 0x7f460c071480 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.230+0000 7f4611fad700 1 -- 172.21.15.89:0/4047747564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460c101480 con 0x7f460c071480 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.230+0000 7f4608ff9700 1 -- 172.21.15.89:0/4047747564 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f45fc010d50 con 0x7f460c071480 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.230+0000 7f4608ff9700 1 -- 172.21.15.89:0/4047747564 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45fc01ae20 con 0x7f460c071480 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.230+0000 7f4611fad700 1 -- 172.21.15.89:0/4047747564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460c101910 con 0x7f460c071480 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.230+0000 7f4608ff9700 1 -- 172.21.15.89:0/4047747564 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f45fc02e440 con 0x7f460c071480 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.231+0000 7f4611fad700 1 -- 172.21.15.89:0/4047747564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460c04bed0 con 0x7f460c071480 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.231+0000 7f4608ff9700 1 --2- 172.21.15.89:0/4047747564 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f45f8036770 0x7f45f8038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.231+0000 7f4608ff9700 1 -- 172.21.15.89:0/4047747564 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f45fc04a4f0 con 0x7f460c071480 2024-02-10T06:08:21.886 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.231+0000 7f4602ffd700 1 --2- 172.21.15.89:0/4047747564 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f45f8036770 0x7f45f8038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.231+0000 7f4602ffd700 1 --2- 172.21.15.89:0/4047747564 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f45f8036770 0x7f45f8038c20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f45f4009990 tx=0x7f45f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.235+0000 7f4608ff9700 1 -- 172.21.15.89:0/4047747564 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f45fc018760 con 0x7f460c071480 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.434+0000 7f4611fad700 1 -- 172.21.15.89:0/4047747564 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f460c102090 con 0x7f45f8036770 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.438+0000 7f4608ff9700 1 -- 172.21.15.89:0/4047747564 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f460c102090 con 0x7f45f8036770 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.440+0000 7f4611fad700 1 -- 172.21.15.89:0/4047747564 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f45f8036770 msgr2=0x7f45f8038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.440+0000 7f4611fad700 1 --2- 172.21.15.89:0/4047747564 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f45f8036770 0x7f45f8038c20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f45f4009990 tx=0x7f45f4006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.440+0000 7f4611fad700 1 -- 172.21.15.89:0/4047747564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 msgr2=0x7f460c102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.440+0000 7f4611fad700 1 --2- 172.21.15.89:0/4047747564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c102e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f45fc003050 tx=0x7f45fc009250 comp rx=0 tx=0).stop 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.440+0000 7f4611fad700 1 -- 172.21.15.89:0/4047747564 shutdown_connections 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.440+0000 7f4611fad700 1 --2- 172.21.15.89:0/4047747564 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f45f8036770 0x7f45f8038c20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.440+0000 7f4611fad700 1 --2- 172.21.15.89:0/4047747564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f460c071480 0x7f460c102e00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.440+0000 7f4611fad700 1 -- 172.21.15.89:0/4047747564 >> 172.21.15.89:0/4047747564 conn(0x7f460c0fcdd0 msgr2=0x7f460c0fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.440+0000 7f4611fad700 1 -- 172.21.15.89:0/4047747564 shutdown_connections 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:20.440+0000 7f4611fad700 1 -- 172.21.15.89:0/4047747564 wait complete. 2024-02-10T06:08:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying crash service with default placement... 2024-02-10T06:08:22.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:22 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:22.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:22 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:23.308 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.297+0000 7fb05f666700 1 Processor -- start 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.298+0000 7fb05f666700 1 -- start start 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.298+0000 7fb05f666700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb058102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.298+0000 7fb05f666700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581030c0 con 0x7fb058102770 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.298+0000 7fb05d402700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb058102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.298+0000 7fb05d402700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb058102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41698/0 (socket says 172.21.15.89:41698) 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.298+0000 7fb05d402700 1 -- 172.21.15.89:0/2017046260 learned_addr learned my addr 172.21.15.89:0/2017046260 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.298+0000 7fb05d402700 1 -- 172.21.15.89:0/2017046260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb058103200 con 0x7fb058102770 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.299+0000 7fb05d402700 1 --2- 172.21.15.89:0/2017046260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb058102b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb0480098e0 tx=0x7fb048009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63af5ced8d4013a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.299+0000 7fb04ffff700 1 -- 172.21.15.89:0/2017046260 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb04800b860 con 0x7fb058102770 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.299+0000 7fb04ffff700 1 -- 172.21.15.89:0/2017046260 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb04800be90 con 0x7fb058102770 2024-02-10T06:08:23.309 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.299+0000 7fb04ffff700 1 -- 172.21.15.89:0/2017046260 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0480103c0 con 0x7fb058102770 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.299+0000 7fb05f666700 1 -- 172.21.15.89:0/2017046260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 msgr2=0x7fb058102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.299+0000 7fb05f666700 1 --2- 172.21.15.89:0/2017046260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb058102b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb0480098e0 tx=0x7fb048009250 comp rx=0 tx=0).stop 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.299+0000 7fb05f666700 1 -- 172.21.15.89:0/2017046260 shutdown_connections 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.299+0000 7fb05f666700 1 --2- 172.21.15.89:0/2017046260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb058102b80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.299+0000 7fb05f666700 1 -- 172.21.15.89:0/2017046260 >> 172.21.15.89:0/2017046260 conn(0x7fb0580fded0 msgr2=0x7fb058100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.300+0000 7fb05f666700 1 -- 172.21.15.89:0/2017046260 shutdown_connections 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.300+0000 7fb05f666700 1 -- 172.21.15.89:0/2017046260 wait complete. 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.300+0000 7fb05f666700 1 Processor -- start 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.300+0000 7fb05f666700 1 -- start start 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.300+0000 7fb05f666700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb0581115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.300+0000 7fb05f666700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb058111af0 con 0x7fb058102770 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.300+0000 7fb05d402700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb0581115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:23.310 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.300+0000 7fb05d402700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb0581115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41700/0 (socket says 172.21.15.89:41700) 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.300+0000 7fb05d402700 1 -- 172.21.15.89:0/1317553258 learned_addr learned my addr 172.21.15.89:0/1317553258 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.300+0000 7fb05d402700 1 -- 172.21.15.89:0/1317553258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0480094f0 con 0x7fb058102770 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.301+0000 7fb05d402700 1 --2- 172.21.15.89:0/1317553258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb0581115b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb048011680 tx=0x7fb048009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.301+0000 7fb04dffb700 1 -- 172.21.15.89:0/1317553258 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb048010890 con 0x7fb058102770 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.301+0000 7fb04dffb700 1 -- 172.21.15.89:0/1317553258 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb048010ec0 con 0x7fb058102770 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.301+0000 7fb05f666700 1 -- 172.21.15.89:0/1317553258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb058111cf0 con 0x7fb058102770 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.301+0000 7fb04dffb700 1 -- 172.21.15.89:0/1317553258 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0480243f0 con 0x7fb058102770 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.301+0000 7fb05f666700 1 -- 172.21.15.89:0/1317553258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb058112990 con 0x7fb058102770 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.302+0000 7fb04dffb700 1 -- 172.21.15.89:0/1317553258 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb04802d3f0 con 0x7fb058102770 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.302+0000 7fb04dffb700 1 --2- 172.21.15.89:0/1317553258 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb044036790 0x7fb044038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.302+0000 7fb05f666700 1 -- 172.21.15.89:0/1317553258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05805f5c0 con 0x7fb058102770 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.302+0000 7fb04dffb700 1 -- 172.21.15.89:0/1317553258 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb04804a490 con 0x7fb058102770 2024-02-10T06:08:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.302+0000 7fb05cc01700 1 --2- 172.21.15.89:0/1317553258 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb044036790 0x7fb044038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.303+0000 7fb05cc01700 1 --2- 172.21.15.89:0/1317553258 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb044036790 0x7fb044038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb054009990 tx=0x7fb054006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.307+0000 7fb04dffb700 1 -- 172.21.15.89:0/1317553258 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb04802b8e0 con 0x7fb058102770 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.514+0000 7fb05f666700 1 -- 172.21.15.89:0/1317553258 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0581001b0 con 0x7fb044036790 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.518+0000 7fb04dffb700 1 -- 172.21.15.89:0/1317553258 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fb0581001b0 con 0x7fb044036790 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.520+0000 7fb05f666700 1 -- 172.21.15.89:0/1317553258 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb044036790 msgr2=0x7fb044038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.520+0000 7fb05f666700 1 --2- 172.21.15.89:0/1317553258 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb044036790 0x7fb044038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb054009990 tx=0x7fb054006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.520+0000 7fb05f666700 1 -- 172.21.15.89:0/1317553258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 msgr2=0x7fb0581115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.520+0000 7fb05f666700 1 --2- 172.21.15.89:0/1317553258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb0581115b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb048011680 tx=0x7fb048009250 comp rx=0 tx=0).stop 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.520+0000 7fb05f666700 1 -- 172.21.15.89:0/1317553258 shutdown_connections 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.520+0000 7fb05f666700 1 --2- 172.21.15.89:0/1317553258 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fb044036790 0x7fb044038c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.520+0000 7fb05f666700 1 --2- 172.21.15.89:0/1317553258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb058102770 0x7fb0581115b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.520+0000 7fb05f666700 1 -- 172.21.15.89:0/1317553258 >> 172.21.15.89:0/1317553258 conn(0x7fb0580fded0 msgr2=0x7fb0580ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.520+0000 7fb05f666700 1 -- 172.21.15.89:0/1317553258 shutdown_connections 2024-02-10T06:08:23.313 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:22.520+0000 7fb05f666700 1 -- 172.21.15.89:0/1317553258 wait complete. 2024-02-10T06:08:23.313 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying prometheus service with default placement... 2024-02-10T06:08:23.962 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:23 smithi089 ceph-mon[29217]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:23.963 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:23 smithi089 ceph-mon[29217]: Saving service crash spec with placement * 2024-02-10T06:08:24.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-02-10T06:08:24.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.707+0000 7fc6a21ab700 1 Processor -- start 2024-02-10T06:08:24.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.707+0000 7fc6a21ab700 1 -- start start 2024-02-10T06:08:24.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.707+0000 7fc6a21ab700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c077210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:24.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.707+0000 7fc6a21ab700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c079110 con 0x7fc69c078d00 2024-02-10T06:08:24.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.707+0000 7fc69b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c077210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:24.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.707+0000 7fc69b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c077210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41712/0 (socket says 172.21.15.89:41712) 2024-02-10T06:08:24.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.707+0000 7fc69b7fe700 1 -- 172.21.15.89:0/1527779921 learned_addr learned my addr 172.21.15.89:0/1527779921 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:24.462 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.707+0000 7fc69b7fe700 1 -- 172.21.15.89:0/1527779921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c077750 con 0x7fc69c078d00 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.707+0000 7fc69b7fe700 1 --2- 172.21.15.89:0/1527779921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c077210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6900098e0 tx=0x7fc690009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95bd71cf3a3c6e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.707+0000 7fc69a7fc700 1 -- 172.21.15.89:0/1527779921 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc69000b860 con 0x7fc69c078d00 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.708+0000 7fc69a7fc700 1 -- 172.21.15.89:0/1527779921 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc69000be90 con 0x7fc69c078d00 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.708+0000 7fc69a7fc700 1 -- 172.21.15.89:0/1527779921 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6900103c0 con 0x7fc69c078d00 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.708+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1527779921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 msgr2=0x7fc69c077210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.708+0000 7fc6a21ab700 1 --2- 172.21.15.89:0/1527779921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c077210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6900098e0 tx=0x7fc690009250 comp rx=0 tx=0).stop 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.708+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1527779921 shutdown_connections 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.708+0000 7fc6a21ab700 1 --2- 172.21.15.89:0/1527779921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c077210 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.708+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1527779921 >> 172.21.15.89:0/1527779921 conn(0x7fc69c0fded0 msgr2=0x7fc69c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.708+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1527779921 shutdown_connections 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.708+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1527779921 wait complete. 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.709+0000 7fc6a21ab700 1 Processor -- start 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.709+0000 7fc6a21ab700 1 -- start start 2024-02-10T06:08:24.463 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.709+0000 7fc6a21ab700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c113800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.709+0000 7fc6a21ab700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c113d40 con 0x7fc69c078d00 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.709+0000 7fc69b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c113800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.709+0000 7fc69b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c113800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41728/0 (socket says 172.21.15.89:41728) 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.709+0000 7fc69b7fe700 1 -- 172.21.15.89:0/1968545600 learned_addr learned my addr 172.21.15.89:0/1968545600 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.709+0000 7fc69b7fe700 1 -- 172.21.15.89:0/1968545600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6900094f0 con 0x7fc69c078d00 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.709+0000 7fc69b7fe700 1 --2- 172.21.15.89:0/1968545600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c113800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc6900094c0 tx=0x7fc690009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.710+0000 7fc683fff700 1 -- 172.21.15.89:0/1968545600 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc690010890 con 0x7fc69c078d00 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.710+0000 7fc683fff700 1 -- 172.21.15.89:0/1968545600 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc690010ec0 con 0x7fc69c078d00 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.710+0000 7fc683fff700 1 -- 172.21.15.89:0/1968545600 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6900243f0 con 0x7fc69c078d00 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.710+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1968545600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69c113f40 con 0x7fc69c078d00 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.710+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1968545600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69c114be0 con 0x7fc69c078d00 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.710+0000 7fc683fff700 1 -- 172.21.15.89:0/1968545600 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fc69002d3f0 con 0x7fc69c078d00 2024-02-10T06:08:24.464 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.711+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1968545600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69c05f5c0 con 0x7fc69c078d00 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.711+0000 7fc683fff700 1 --2- 172.21.15.89:0/1968545600 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fc6840367e0 0x7fc684038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.711+0000 7fc683fff700 1 -- 172.21.15.89:0/1968545600 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc69004a3e0 con 0x7fc69c078d00 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.711+0000 7fc69affd700 1 --2- 172.21.15.89:0/1968545600 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fc6840367e0 0x7fc684038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.711+0000 7fc69affd700 1 --2- 172.21.15.89:0/1968545600 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fc6840367e0 0x7fc684038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009990 tx=0x7fc68c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.715+0000 7fc683fff700 1 -- 172.21.15.89:0/1968545600 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc6900193c0 con 0x7fc69c078d00 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.912+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1968545600 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc69c100210 con 0x7fc6840367e0 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.916+0000 7fc683fff700 1 -- 172.21.15.89:0/1968545600 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fc69c100210 con 0x7fc6840367e0 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.918+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1968545600 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fc6840367e0 msgr2=0x7fc684038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.918+0000 7fc6a21ab700 1 --2- 172.21.15.89:0/1968545600 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fc6840367e0 0x7fc684038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009990 tx=0x7fc68c006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.918+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1968545600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 msgr2=0x7fc69c113800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.918+0000 7fc6a21ab700 1 --2- 172.21.15.89:0/1968545600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c113800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc6900094c0 tx=0x7fc690009250 comp rx=0 tx=0).stop 2024-02-10T06:08:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.918+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1968545600 shutdown_connections 2024-02-10T06:08:24.466 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.918+0000 7fc6a21ab700 1 --2- 172.21.15.89:0/1968545600 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fc6840367e0 0x7fc684038c90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:24.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.918+0000 7fc6a21ab700 1 --2- 172.21.15.89:0/1968545600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc69c078d00 0x7fc69c113800 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:24.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.918+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1968545600 >> 172.21.15.89:0/1968545600 conn(0x7fc69c0fded0 msgr2=0x7fc69c0ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:24.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.918+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1968545600 shutdown_connections 2024-02-10T06:08:24.467 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:23.918+0000 7fc6a21ab700 1 -- 172.21.15.89:0/1968545600 wait complete. 2024-02-10T06:08:24.467 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying grafana service with default placement... 2024-02-10T06:08:25.063 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:24 smithi089 ceph-mon[29217]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:25.063 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:24 smithi089 ceph-mon[29217]: Saving service prometheus spec with placement count:1 2024-02-10T06:08:25.063 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:24 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:25.063 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:24 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:25.635 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-02-10T06:08:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.844+0000 7f8eb913f700 1 Processor -- start 2024-02-10T06:08:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.844+0000 7f8eb913f700 1 -- start start 2024-02-10T06:08:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.844+0000 7f8eb913f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4102b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.844+0000 7f8eb913f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb41030d0 con 0x7f8eb4102780 2024-02-10T06:08:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.844+0000 7f8eb2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.844+0000 7f8eb2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41738/0 (socket says 172.21.15.89:41738) 2024-02-10T06:08:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.844+0000 7f8eb2d9d700 1 -- 172.21.15.89:0/2134291031 learned_addr learned my addr 172.21.15.89:0/2134291031 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.844+0000 7f8eb2d9d700 1 -- 172.21.15.89:0/2134291031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4103210 con 0x7f8eb4102780 2024-02-10T06:08:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.844+0000 7f8eb2d9d700 1 --2- 172.21.15.89:0/2134291031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4102b90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80098e0 tx=0x7f8ea8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f9424c70fd187d server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.845+0000 7f8eb1d9b700 1 -- 172.21.15.89:0/2134291031 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ea800b860 con 0x7f8eb4102780 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.845+0000 7f8eb1d9b700 1 -- 172.21.15.89:0/2134291031 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ea800be90 con 0x7f8eb4102780 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.845+0000 7f8eb1d9b700 1 -- 172.21.15.89:0/2134291031 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ea80103c0 con 0x7f8eb4102780 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.846+0000 7f8eb913f700 1 -- 172.21.15.89:0/2134291031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 msgr2=0x7f8eb4102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.846+0000 7f8eb913f700 1 --2- 172.21.15.89:0/2134291031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4102b90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80098e0 tx=0x7f8ea8009250 comp rx=0 tx=0).stop 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.846+0000 7f8eb913f700 1 -- 172.21.15.89:0/2134291031 shutdown_connections 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.846+0000 7f8eb913f700 1 --2- 172.21.15.89:0/2134291031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4102b90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.846+0000 7f8eb913f700 1 -- 172.21.15.89:0/2134291031 >> 172.21.15.89:0/2134291031 conn(0x7f8eb40fdf00 msgr2=0x7f8eb4100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.846+0000 7f8eb913f700 1 -- 172.21.15.89:0/2134291031 shutdown_connections 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.846+0000 7f8eb913f700 1 -- 172.21.15.89:0/2134291031 wait complete. 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.846+0000 7f8eb913f700 1 Processor -- start 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.846+0000 7f8eb913f700 1 -- start start 2024-02-10T06:08:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.846+0000 7f8eb913f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.847+0000 7f8eb913f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb4077190 con 0x7f8eb4102780 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.847+0000 7f8eb2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.847+0000 7f8eb2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41740/0 (socket says 172.21.15.89:41740) 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.847+0000 7f8eb2d9d700 1 -- 172.21.15.89:0/1343281112 learned_addr learned my addr 172.21.15.89:0/1343281112 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.847+0000 7f8eb2d9d700 1 -- 172.21.15.89:0/1343281112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea80094f0 con 0x7f8eb4102780 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.847+0000 7f8eb2d9d700 1 --2- 172.21.15.89:0/1343281112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4078d10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8011680 tx=0x7f8ea8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.847+0000 7f8e9b7fe700 1 -- 172.21.15.89:0/1343281112 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ea8010890 con 0x7f8eb4102780 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.847+0000 7f8e9b7fe700 1 -- 172.21.15.89:0/1343281112 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ea8010ec0 con 0x7f8eb4102780 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.847+0000 7f8eb913f700 1 -- 172.21.15.89:0/1343281112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb4077390 con 0x7f8eb4102780 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.847+0000 7f8e9b7fe700 1 -- 172.21.15.89:0/1343281112 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ea80243f0 con 0x7f8eb4102780 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.847+0000 7f8eb913f700 1 -- 172.21.15.89:0/1343281112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb4077820 con 0x7f8eb4102780 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.848+0000 7f8e9b7fe700 1 -- 172.21.15.89:0/1343281112 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f8ea802d3f0 con 0x7f8eb4102780 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.848+0000 7f8e9b7fe700 1 --2- 172.21.15.89:0/1343281112 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f8e9c0367e0 0x7f8e9c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.848+0000 7f8e9b7fe700 1 -- 172.21.15.89:0/1343281112 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8ea801a8c0 con 0x7f8eb4102780 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.848+0000 7f8eb259c700 1 --2- 172.21.15.89:0/1343281112 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f8e9c0367e0 0x7f8e9c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.849+0000 7f8eb913f700 1 -- 172.21.15.89:0/1343281112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb405f5c0 con 0x7f8eb4102780 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.849+0000 7f8eb259c700 1 --2- 172.21.15.89:0/1343281112 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f8e9c0367e0 0x7f8e9c038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009990 tx=0x7f8ea4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:24.853+0000 7f8e9b7fe700 1 -- 172.21.15.89:0/1343281112 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8ea802b8e0 con 0x7f8eb4102780 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.055+0000 7f8eb913f700 1 -- 172.21.15.89:0/1343281112 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f8eb4077c50 con 0x7f8e9c0367e0 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.060+0000 7f8e9b7fe700 1 -- 172.21.15.89:0/1343281112 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8eb4077c50 con 0x7f8e9c0367e0 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.062+0000 7f8eb913f700 1 -- 172.21.15.89:0/1343281112 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f8e9c0367e0 msgr2=0x7f8e9c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.062+0000 7f8eb913f700 1 --2- 172.21.15.89:0/1343281112 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f8e9c0367e0 0x7f8e9c038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009990 tx=0x7f8ea4006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.062+0000 7f8eb913f700 1 -- 172.21.15.89:0/1343281112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 msgr2=0x7f8eb4078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.062+0000 7f8eb913f700 1 --2- 172.21.15.89:0/1343281112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4078d10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8011680 tx=0x7f8ea8009250 comp rx=0 tx=0).stop 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.063+0000 7f8eb913f700 1 -- 172.21.15.89:0/1343281112 shutdown_connections 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.063+0000 7f8eb913f700 1 --2- 172.21.15.89:0/1343281112 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f8e9c0367e0 0x7f8e9c038c90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.063+0000 7f8eb913f700 1 --2- 172.21.15.89:0/1343281112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8eb4102780 0x7f8eb4078d10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:25.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.063+0000 7f8eb913f700 1 -- 172.21.15.89:0/1343281112 >> 172.21.15.89:0/1343281112 conn(0x7f8eb40fdf00 msgr2=0x7f8eb40ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:25.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.063+0000 7f8eb913f700 1 -- 172.21.15.89:0/1343281112 shutdown_connections 2024-02-10T06:08:25.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:25.063+0000 7f8eb913f700 1 -- 172.21.15.89:0/1343281112 wait complete. 2024-02-10T06:08:25.645 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying node-exporter service with default placement... 2024-02-10T06:08:26.245 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:26 smithi089 ceph-mon[29217]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:26.245 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:26 smithi089 ceph-mon[29217]: Saving service grafana spec with placement count:1 2024-02-10T06:08:26.245 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:26 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:26.245 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:26 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:26.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-02-10T06:08:26.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.086+0000 7fe3082b9700 1 Processor -- start 2024-02-10T06:08:26.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.087+0000 7fe3082b9700 1 -- start start 2024-02-10T06:08:26.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.087+0000 7fe3082b9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe3000771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:26.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.087+0000 7fe3082b9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000790d0 con 0x7fe300078cc0 2024-02-10T06:08:26.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.087+0000 7fe306055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe3000771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:26.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.087+0000 7fe306055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe3000771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41748/0 (socket says 172.21.15.89:41748) 2024-02-10T06:08:26.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.087+0000 7fe306055700 1 -- 172.21.15.89:0/3688748115 learned_addr learned my addr 172.21.15.89:0/3688748115 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:26.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.087+0000 7fe306055700 1 -- 172.21.15.89:0/3688748115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe300077710 con 0x7fe300078cc0 2024-02-10T06:08:26.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.088+0000 7fe306055700 1 --2- 172.21.15.89:0/3688748115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe3000771d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0098e0 tx=0x7fe2fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cfbd3307e02ce7 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:26.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.088+0000 7fe305053700 1 -- 172.21.15.89:0/3688748115 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2fc00b860 con 0x7fe300078cc0 2024-02-10T06:08:26.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.088+0000 7fe305053700 1 -- 172.21.15.89:0/3688748115 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2fc00be90 con 0x7fe300078cc0 2024-02-10T06:08:26.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.088+0000 7fe305053700 1 -- 172.21.15.89:0/3688748115 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2fc0103c0 con 0x7fe300078cc0 2024-02-10T06:08:26.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.089+0000 7fe3082b9700 1 -- 172.21.15.89:0/3688748115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 msgr2=0x7fe3000771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:26.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.089+0000 7fe3082b9700 1 --2- 172.21.15.89:0/3688748115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe3000771d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0098e0 tx=0x7fe2fc009250 comp rx=0 tx=0).stop 2024-02-10T06:08:26.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.089+0000 7fe3082b9700 1 -- 172.21.15.89:0/3688748115 shutdown_connections 2024-02-10T06:08:26.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.089+0000 7fe3082b9700 1 --2- 172.21.15.89:0/3688748115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe3000771d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.089+0000 7fe3082b9700 1 -- 172.21.15.89:0/3688748115 >> 172.21.15.89:0/3688748115 conn(0x7fe3000fdf00 msgr2=0x7fe300100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.089+0000 7fe3082b9700 1 -- 172.21.15.89:0/3688748115 shutdown_connections 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.089+0000 7fe3082b9700 1 -- 172.21.15.89:0/3688748115 wait complete. 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.090+0000 7fe3082b9700 1 Processor -- start 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.090+0000 7fe3082b9700 1 -- start start 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.090+0000 7fe3082b9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe30011bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.090+0000 7fe3082b9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30011c530 con 0x7fe300078cc0 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.090+0000 7fe306055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe30011bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.090+0000 7fe306055700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe30011bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41754/0 (socket says 172.21.15.89:41754) 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.090+0000 7fe306055700 1 -- 172.21.15.89:0/1377952691 learned_addr learned my addr 172.21.15.89:0/1377952691 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.090+0000 7fe306055700 1 -- 172.21.15.89:0/1377952691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc0094f0 con 0x7fe300078cc0 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.091+0000 7fe306055700 1 --2- 172.21.15.89:0/1377952691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe30011bff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc006d40 tx=0x7fe2fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.091+0000 7fe2f6ffd700 1 -- 172.21.15.89:0/1377952691 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2fc010840 con 0x7fe300078cc0 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.091+0000 7fe3082b9700 1 -- 172.21.15.89:0/1377952691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30011c730 con 0x7fe300078cc0 2024-02-10T06:08:26.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.091+0000 7fe2f6ffd700 1 -- 172.21.15.89:0/1377952691 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2fc010e70 con 0x7fe300078cc0 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.091+0000 7fe2f6ffd700 1 -- 172.21.15.89:0/1377952691 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2fc024ea0 con 0x7fe300078cc0 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.091+0000 7fe3082b9700 1 -- 172.21.15.89:0/1377952691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30011d3d0 con 0x7fe300078cc0 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.092+0000 7fe3082b9700 1 -- 172.21.15.89:0/1377952691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30005f5c0 con 0x7fe300078cc0 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.092+0000 7fe2f6ffd700 1 -- 172.21.15.89:0/1377952691 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe2fc01a510 con 0x7fe300078cc0 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.092+0000 7fe2f6ffd700 1 --2- 172.21.15.89:0/1377952691 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fe2ec036740 0x7fe2ec038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.092+0000 7fe2f6ffd700 1 -- 172.21.15.89:0/1377952691 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe2fc04a740 con 0x7fe300078cc0 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.093+0000 7fe305854700 1 --2- 172.21.15.89:0/1377952691 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fe2ec036740 0x7fe2ec038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.093+0000 7fe305854700 1 --2- 172.21.15.89:0/1377952691 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fe2ec036740 0x7fe2ec038bf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009990 tx=0x7fe2f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.101+0000 7fe2f6ffd700 1 -- 172.21.15.89:0/1377952691 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe2fc018940 con 0x7fe300078cc0 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.320+0000 7fe3082b9700 1 -- 172.21.15.89:0/1377952691 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fe300100200 con 0x7fe2ec036740 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.325+0000 7fe2f6ffd700 1 -- 172.21.15.89:0/1377952691 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fe300100200 con 0x7fe2ec036740 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.327+0000 7fe3082b9700 1 -- 172.21.15.89:0/1377952691 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fe2ec036740 msgr2=0x7fe2ec038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.327+0000 7fe3082b9700 1 --2- 172.21.15.89:0/1377952691 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fe2ec036740 0x7fe2ec038bf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009990 tx=0x7fe2f0006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:26.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.327+0000 7fe3082b9700 1 -- 172.21.15.89:0/1377952691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 msgr2=0x7fe30011bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:26.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.327+0000 7fe3082b9700 1 --2- 172.21.15.89:0/1377952691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe30011bff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc006d40 tx=0x7fe2fc009250 comp rx=0 tx=0).stop 2024-02-10T06:08:26.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.327+0000 7fe3082b9700 1 -- 172.21.15.89:0/1377952691 shutdown_connections 2024-02-10T06:08:26.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.327+0000 7fe3082b9700 1 --2- 172.21.15.89:0/1377952691 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fe2ec036740 0x7fe2ec038bf0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:26.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.327+0000 7fe3082b9700 1 --2- 172.21.15.89:0/1377952691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300078cc0 0x7fe30011bff0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:26.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.327+0000 7fe3082b9700 1 -- 172.21.15.89:0/1377952691 >> 172.21.15.89:0/1377952691 conn(0x7fe3000fdf00 msgr2=0x7fe3000ff640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:26.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.327+0000 7fe3082b9700 1 -- 172.21.15.89:0/1377952691 shutdown_connections 2024-02-10T06:08:26.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:26.327+0000 7fe3082b9700 1 -- 172.21.15.89:0/1377952691 wait complete. 2024-02-10T06:08:26.820 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying alertmanager service with default placement... 2024-02-10T06:08:27.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:27 smithi089 ceph-mon[29217]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:27.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:27 smithi089 ceph-mon[29217]: Saving service node-exporter spec with placement * 2024-02-10T06:08:27.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:27 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:28.099 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-02-10T06:08:28.099 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.255+0000 7f6653716700 1 Processor -- start 2024-02-10T06:08:28.099 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.256+0000 7f6653716700 1 -- start start 2024-02-10T06:08:28.099 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.256+0000 7f6653716700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:28.099 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.256+0000 7f6653716700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c1058c0 con 0x7f664c076c20 2024-02-10T06:08:28.099 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.256+0000 7f66514b2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:28.099 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.256+0000 7f66514b2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41760/0 (socket says 172.21.15.89:41760) 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.256+0000 7f66514b2700 1 -- 172.21.15.89:0/3061433441 learned_addr learned my addr 172.21.15.89:0/3061433441 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.256+0000 7f66514b2700 1 -- 172.21.15.89:0/3061433441 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c105a00 con 0x7f664c076c20 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.256+0000 7f66514b2700 1 --2- 172.21.15.89:0/3061433441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c105380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f663c0098e0 tx=0x7f663c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104cadb6f30f2e45 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.257+0000 7f6643fff700 1 -- 172.21.15.89:0/3061433441 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f663c00b860 con 0x7f664c076c20 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.257+0000 7f6643fff700 1 -- 172.21.15.89:0/3061433441 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f663c00be90 con 0x7f664c076c20 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.257+0000 7f6643fff700 1 -- 172.21.15.89:0/3061433441 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f663c0103c0 con 0x7f664c076c20 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.257+0000 7f6653716700 1 -- 172.21.15.89:0/3061433441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 msgr2=0x7f664c105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.257+0000 7f6653716700 1 --2- 172.21.15.89:0/3061433441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c105380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f663c0098e0 tx=0x7f663c009250 comp rx=0 tx=0).stop 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.257+0000 7f6653716700 1 -- 172.21.15.89:0/3061433441 shutdown_connections 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.257+0000 7f6653716700 1 --2- 172.21.15.89:0/3061433441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c105380 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.257+0000 7f6653716700 1 -- 172.21.15.89:0/3061433441 >> 172.21.15.89:0/3061433441 conn(0x7f664c0fcdd0 msgr2=0x7f664c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.257+0000 7f6653716700 1 -- 172.21.15.89:0/3061433441 shutdown_connections 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.257+0000 7f6653716700 1 -- 172.21.15.89:0/3061433441 wait complete. 2024-02-10T06:08:28.100 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.258+0000 7f6653716700 1 Processor -- start 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.258+0000 7f6653716700 1 -- start start 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.258+0000 7f6653716700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c111870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.258+0000 7f6653716700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c111db0 con 0x7f664c076c20 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.258+0000 7f66514b2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c111870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.258+0000 7f66514b2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c111870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41766/0 (socket says 172.21.15.89:41766) 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.258+0000 7f66514b2700 1 -- 172.21.15.89:0/1166154401 learned_addr learned my addr 172.21.15.89:0/1166154401 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.258+0000 7f66514b2700 1 -- 172.21.15.89:0/1166154401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c0094f0 con 0x7f664c076c20 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.259+0000 7f66514b2700 1 --2- 172.21.15.89:0/1166154401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c111870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f663c0090b0 tx=0x7f663c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.259+0000 7f6641ffb700 1 -- 172.21.15.89:0/1166154401 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f663c0118f0 con 0x7f664c076c20 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.259+0000 7f6641ffb700 1 -- 172.21.15.89:0/1166154401 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f663c010d10 con 0x7f664c076c20 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.259+0000 7f6653716700 1 -- 172.21.15.89:0/1166154401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c111fb0 con 0x7f664c076c20 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.259+0000 7f6641ffb700 1 -- 172.21.15.89:0/1166154401 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f663c024e70 con 0x7f664c076c20 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.259+0000 7f6653716700 1 -- 172.21.15.89:0/1166154401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c112c50 con 0x7f664c076c20 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.260+0000 7f6641ffb700 1 -- 172.21.15.89:0/1166154401 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f663c01a550 con 0x7f664c076c20 2024-02-10T06:08:28.101 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.260+0000 7f6641ffb700 1 --2- 172.21.15.89:0/1166154401 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f6638036740 0x7f6638038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.260+0000 7f6641ffb700 1 -- 172.21.15.89:0/1166154401 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f663c04a650 con 0x7f664c076c20 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.260+0000 7f6650cb1700 1 --2- 172.21.15.89:0/1166154401 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f6638036740 0x7f6638038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.260+0000 7f6653716700 1 -- 172.21.15.89:0/1166154401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c112440 con 0x7f664c076c20 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.260+0000 7f6650cb1700 1 --2- 172.21.15.89:0/1166154401 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f6638036740 0x7f6638038bf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6648009990 tx=0x7f6648006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.265+0000 7f6641ffb700 1 -- 172.21.15.89:0/1166154401 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f663c02d3f0 con 0x7f664c076c20 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.463+0000 7f6653716700 1 -- 172.21.15.89:0/1166154401 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f664c106850 con 0x7f6638036740 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.469+0000 7f6641ffb700 1 -- 172.21.15.89:0/1166154401 <== mgr.14120 v2:172.21.15.89:6800/1397975713 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f664c106850 con 0x7f6638036740 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.471+0000 7f6653716700 1 -- 172.21.15.89:0/1166154401 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f6638036740 msgr2=0x7f6638038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.471+0000 7f6653716700 1 --2- 172.21.15.89:0/1166154401 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f6638036740 0x7f6638038bf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6648009990 tx=0x7f6648006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.471+0000 7f6653716700 1 -- 172.21.15.89:0/1166154401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 msgr2=0x7f664c111870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.471+0000 7f6653716700 1 --2- 172.21.15.89:0/1166154401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c111870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f663c0090b0 tx=0x7f663c009250 comp rx=0 tx=0).stop 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.471+0000 7f6653716700 1 -- 172.21.15.89:0/1166154401 shutdown_connections 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.471+0000 7f6653716700 1 --2- 172.21.15.89:0/1166154401 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f6638036740 0x7f6638038bf0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:28.102 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.471+0000 7f6653716700 1 --2- 172.21.15.89:0/1166154401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f664c076c20 0x7f664c111870 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:28.103 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.471+0000 7f6653716700 1 -- 172.21.15.89:0/1166154401 >> 172.21.15.89:0/1166154401 conn(0x7f664c0fcdd0 msgr2=0x7f664c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:28.103 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.472+0000 7f6653716700 1 -- 172.21.15.89:0/1166154401 shutdown_connections 2024-02-10T06:08:28.103 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:27.472+0000 7f6653716700 1 -- 172.21.15.89:0/1166154401 wait complete. 2024-02-10T06:08:28.694 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:28 smithi089 ceph-mon[29217]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:28.694 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:28 smithi089 ceph-mon[29217]: Saving service alertmanager spec with placement count:1 2024-02-10T06:08:28.694 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:28 smithi089 ceph-mon[29217]: from='mgr.14120 172.21.15.89:0/1812246633' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:29.300 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.498+0000 7fd568fa9700 1 Processor -- start 2024-02-10T06:08:29.300 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.498+0000 7fd568fa9700 1 -- start start 2024-02-10T06:08:29.300 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.499+0000 7fd568fa9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd564103050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.499+0000 7fd568fa9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd564103590 con 0x7fd564102c40 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.499+0000 7fd56259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd564103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.499+0000 7fd56259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd564103050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57566/0 (socket says 172.21.15.89:57566) 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.499+0000 7fd56259c700 1 -- 172.21.15.89:0/11294004 learned_addr learned my addr 172.21.15.89:0/11294004 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.499+0000 7fd56259c700 1 -- 172.21.15.89:0/11294004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5641036d0 con 0x7fd564102c40 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.499+0000 7fd56259c700 1 --2- 172.21.15.89:0/11294004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd564103050 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0098e0 tx=0x7fd55c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbad6a653daf86e server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.500+0000 7fd56159a700 1 -- 172.21.15.89:0/11294004 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd55c00b860 con 0x7fd564102c40 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.500+0000 7fd56159a700 1 -- 172.21.15.89:0/11294004 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd55c00be90 con 0x7fd564102c40 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.500+0000 7fd56159a700 1 -- 172.21.15.89:0/11294004 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd55c0103c0 con 0x7fd564102c40 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.500+0000 7fd568fa9700 1 -- 172.21.15.89:0/11294004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 msgr2=0x7fd564103050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.500+0000 7fd568fa9700 1 --2- 172.21.15.89:0/11294004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd564103050 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0098e0 tx=0x7fd55c009250 comp rx=0 tx=0).stop 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.500+0000 7fd568fa9700 1 -- 172.21.15.89:0/11294004 shutdown_connections 2024-02-10T06:08:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.500+0000 7fd568fa9700 1 --2- 172.21.15.89:0/11294004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd564103050 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.500+0000 7fd568fa9700 1 -- 172.21.15.89:0/11294004 >> 172.21.15.89:0/11294004 conn(0x7fd5640fc580 msgr2=0x7fd5640fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.500+0000 7fd568fa9700 1 -- 172.21.15.89:0/11294004 shutdown_connections 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.500+0000 7fd568fa9700 1 -- 172.21.15.89:0/11294004 wait complete. 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.501+0000 7fd568fa9700 1 Processor -- start 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.501+0000 7fd568fa9700 1 -- start start 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.501+0000 7fd568fa9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd5641145b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.501+0000 7fd568fa9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd564101540 con 0x7fd564102c40 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.501+0000 7fd56259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd5641145b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.501+0000 7fd56259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd5641145b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57578/0 (socket says 172.21.15.89:57578) 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.501+0000 7fd56259c700 1 -- 172.21.15.89:0/3813258106 learned_addr learned my addr 172.21.15.89:0/3813258106 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.501+0000 7fd56259c700 1 -- 172.21.15.89:0/3813258106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c0094f0 con 0x7fd564102c40 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.502+0000 7fd56259c700 1 --2- 172.21.15.89:0/3813258106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd5641145b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0105b0 tx=0x7fd55c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.502+0000 7fd552ffd700 1 -- 172.21.15.89:0/3813258106 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd55c010680 con 0x7fd564102c40 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.502+0000 7fd552ffd700 1 -- 172.21.15.89:0/3813258106 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd55c010cb0 con 0x7fd564102c40 2024-02-10T06:08:29.302 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.502+0000 7fd568fa9700 1 -- 172.21.15.89:0/3813258106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd564101740 con 0x7fd564102c40 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.502+0000 7fd552ffd700 1 -- 172.21.15.89:0/3813258106 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd55c01ae00 con 0x7fd564102c40 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.502+0000 7fd568fa9700 1 -- 172.21.15.89:0/3813258106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd564114cb0 con 0x7fd564102c40 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.503+0000 7fd552ffd700 1 -- 172.21.15.89:0/3813258106 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fd55c0193c0 con 0x7fd564102c40 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.503+0000 7fd552ffd700 1 --2- 172.21.15.89:0/3813258106 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd54c036790 0x7fd54c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.503+0000 7fd568fa9700 1 -- 172.21.15.89:0/3813258106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56410a5c0 con 0x7fd564102c40 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.503+0000 7fd552ffd700 1 -- 172.21.15.89:0/3813258106 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd55c049f10 con 0x7fd564102c40 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.503+0000 7fd561d9b700 1 --2- 172.21.15.89:0/3813258106 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd54c036790 0x7fd54c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.503+0000 7fd561d9b700 1 --2- 172.21.15.89:0/3813258106 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd54c036790 0x7fd54c038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd55800ad90 tx=0x7fd558009410 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.507+0000 7fd552ffd700 1 -- 172.21.15.89:0/3813258106 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd55c023710 con 0x7fd564102c40 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.693+0000 7fd568fa9700 1 -- 172.21.15.89:0/3813258106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fd56404bed0 con 0x7fd564102c40 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.697+0000 7fd552ffd700 1 -- 172.21.15.89:0/3813258106 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fd55c023350 con 0x7fd564102c40 2024-02-10T06:08:29.303 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.699+0000 7fd568fa9700 1 -- 172.21.15.89:0/3813258106 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd54c036790 msgr2=0x7fd54c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:29.304 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.699+0000 7fd568fa9700 1 --2- 172.21.15.89:0/3813258106 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd54c036790 0x7fd54c038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd55800ad90 tx=0x7fd558009410 comp rx=0 tx=0).stop 2024-02-10T06:08:29.304 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.699+0000 7fd568fa9700 1 -- 172.21.15.89:0/3813258106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 msgr2=0x7fd5641145b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:29.304 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.699+0000 7fd568fa9700 1 --2- 172.21.15.89:0/3813258106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd5641145b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0105b0 tx=0x7fd55c009250 comp rx=0 tx=0).stop 2024-02-10T06:08:29.304 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.699+0000 7fd568fa9700 1 -- 172.21.15.89:0/3813258106 shutdown_connections 2024-02-10T06:08:29.305 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.699+0000 7fd568fa9700 1 --2- 172.21.15.89:0/3813258106 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fd54c036790 0x7fd54c038c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:29.306 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.699+0000 7fd568fa9700 1 --2- 172.21.15.89:0/3813258106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd564102c40 0x7fd5641145b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:29.306 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.699+0000 7fd568fa9700 1 -- 172.21.15.89:0/3813258106 >> 172.21.15.89:0/3813258106 conn(0x7fd5640fc580 msgr2=0x7fd5640fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:29.306 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.699+0000 7fd568fa9700 1 -- 172.21.15.89:0/3813258106 shutdown_connections 2024-02-10T06:08:29.306 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:28.699+0000 7fd568fa9700 1 -- 172.21.15.89:0/3813258106 wait complete. 2024-02-10T06:08:29.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:29 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3813258106' entity='client.admin' 2024-02-10T06:08:29.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:29 smithi089 ceph-mon[29217]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:08:30.381 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.682+0000 7fbd8ff45700 1 Processor -- start 2024-02-10T06:08:30.381 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.682+0000 7fbd8ff45700 1 -- start start 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.682+0000 7fbd8ff45700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd881031c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.682+0000 7fbd8ff45700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd88103700 con 0x7fbd88102db0 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.683+0000 7fbd8dce1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd881031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.683+0000 7fbd8dce1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd881031c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57592/0 (socket says 172.21.15.89:57592) 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.683+0000 7fbd8dce1700 1 -- 172.21.15.89:0/4228697044 learned_addr learned my addr 172.21.15.89:0/4228697044 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.683+0000 7fbd8dce1700 1 -- 172.21.15.89:0/4228697044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd88103840 con 0x7fbd88102db0 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.683+0000 7fbd8dce1700 1 --2- 172.21.15.89:0/4228697044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd881031c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd840098e0 tx=0x7fbd84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc55a810c5c7e1e server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.683+0000 7fbd8ccdf700 1 -- 172.21.15.89:0/4228697044 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd8400b860 con 0x7fbd88102db0 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.683+0000 7fbd8ccdf700 1 -- 172.21.15.89:0/4228697044 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd8400be90 con 0x7fbd88102db0 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.683+0000 7fbd8ccdf700 1 -- 172.21.15.89:0/4228697044 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd840103c0 con 0x7fbd88102db0 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.684+0000 7fbd8ff45700 1 -- 172.21.15.89:0/4228697044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 msgr2=0x7fbd881031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.684+0000 7fbd8ff45700 1 --2- 172.21.15.89:0/4228697044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd881031c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd840098e0 tx=0x7fbd84009250 comp rx=0 tx=0).stop 2024-02-10T06:08:30.382 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.684+0000 7fbd8ff45700 1 -- 172.21.15.89:0/4228697044 shutdown_connections 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.684+0000 7fbd8ff45700 1 --2- 172.21.15.89:0/4228697044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd881031c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.684+0000 7fbd8ff45700 1 -- 172.21.15.89:0/4228697044 >> 172.21.15.89:0/4228697044 conn(0x7fbd880fc580 msgr2=0x7fbd880fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.684+0000 7fbd8ff45700 1 -- 172.21.15.89:0/4228697044 shutdown_connections 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.684+0000 7fbd8ff45700 1 -- 172.21.15.89:0/4228697044 wait complete. 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.684+0000 7fbd8ff45700 1 Processor -- start 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.684+0000 7fbd8ff45700 1 -- start start 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.684+0000 7fbd8ff45700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd88113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.684+0000 7fbd8ff45700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd88113f90 con 0x7fbd88102db0 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.685+0000 7fbd8dce1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd88113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.685+0000 7fbd8dce1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd88113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57596/0 (socket says 172.21.15.89:57596) 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.685+0000 7fbd8dce1700 1 -- 172.21.15.89:0/659533997 learned_addr learned my addr 172.21.15.89:0/659533997 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.685+0000 7fbd8dce1700 1 -- 172.21.15.89:0/659533997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd840094f0 con 0x7fbd88102db0 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.685+0000 7fbd8dce1700 1 --2- 172.21.15.89:0/659533997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd88113a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbd84006b00 tx=0x7fbd84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:30.383 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.685+0000 7fbd7e7fc700 1 -- 172.21.15.89:0/659533997 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd840109a0 con 0x7fbd88102db0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.685+0000 7fbd7e7fc700 1 -- 172.21.15.89:0/659533997 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd8401a430 con 0x7fbd88102db0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.685+0000 7fbd8ff45700 1 -- 172.21.15.89:0/659533997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd88114190 con 0x7fbd88102db0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.685+0000 7fbd7e7fc700 1 -- 172.21.15.89:0/659533997 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd840245b0 con 0x7fbd88102db0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.685+0000 7fbd8ff45700 1 -- 172.21.15.89:0/659533997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd88114e30 con 0x7fbd88102db0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.686+0000 7fbd7e7fc700 1 -- 172.21.15.89:0/659533997 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fbd840247e0 con 0x7fbd88102db0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.686+0000 7fbd7e7fc700 1 --2- 172.21.15.89:0/659533997 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbd740367e0 0x7fbd74038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.686+0000 7fbd7e7fc700 1 -- 172.21.15.89:0/659533997 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbd8404a620 con 0x7fbd88102db0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.686+0000 7fbd8d4e0700 1 --2- 172.21.15.89:0/659533997 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbd740367e0 0x7fbd74038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.686+0000 7fbd8ff45700 1 -- 172.21.15.89:0/659533997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd8805f5c0 con 0x7fbd88102db0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.686+0000 7fbd8d4e0700 1 --2- 172.21.15.89:0/659533997 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbd740367e0 0x7fbd74038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbd78009990 tx=0x7fbd78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.691+0000 7fbd7e7fc700 1 -- 172.21.15.89:0/659533997 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbd8402e440 con 0x7fbd88102db0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.874+0000 7fbd8ff45700 1 -- 172.21.15.89:0/659533997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fbd8804bed0 con 0x7fbd88102db0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.877+0000 7fbd7e7fc700 1 -- 172.21.15.89:0/659533997 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fbd84018350 con 0x7fbd88102db0 2024-02-10T06:08:30.384 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.879+0000 7fbd8ff45700 1 -- 172.21.15.89:0/659533997 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbd740367e0 msgr2=0x7fbd74038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:30.385 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.879+0000 7fbd8ff45700 1 --2- 172.21.15.89:0/659533997 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbd740367e0 0x7fbd74038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbd78009990 tx=0x7fbd78006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:30.385 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.879+0000 7fbd8ff45700 1 -- 172.21.15.89:0/659533997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 msgr2=0x7fbd88113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:30.385 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.879+0000 7fbd8ff45700 1 --2- 172.21.15.89:0/659533997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd88113a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbd84006b00 tx=0x7fbd84009250 comp rx=0 tx=0).stop 2024-02-10T06:08:30.385 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.879+0000 7fbd8ff45700 1 -- 172.21.15.89:0/659533997 shutdown_connections 2024-02-10T06:08:30.385 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.880+0000 7fbd8ff45700 1 --2- 172.21.15.89:0/659533997 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7fbd740367e0 0x7fbd74038c90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:30.385 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.880+0000 7fbd8ff45700 1 --2- 172.21.15.89:0/659533997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd88102db0 0x7fbd88113a50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:30.385 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.880+0000 7fbd8ff45700 1 -- 172.21.15.89:0/659533997 >> 172.21.15.89:0/659533997 conn(0x7fbd880fc580 msgr2=0x7fbd880fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:30.385 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.880+0000 7fbd8ff45700 1 -- 172.21.15.89:0/659533997 shutdown_connections 2024-02-10T06:08:30.385 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:29.880+0000 7fbd8ff45700 1 -- 172.21.15.89:0/659533997 wait complete. 2024-02-10T06:08:30.385 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling the dashboard module... 2024-02-10T06:08:31.002 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:30 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/659533997' entity='client.admin' 2024-02-10T06:08:32.247 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:31 smithi089 ceph-mon[29217]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:08:32.247 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:31 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2895610682' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T06:08:32.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.755+0000 7f3264339700 1 Processor -- start 2024-02-10T06:08:32.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.755+0000 7f3264339700 1 -- start start 2024-02-10T06:08:32.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.756+0000 7f3264339700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c105360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:32.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.756+0000 7f3264339700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c1058a0 con 0x7f325c076c40 2024-02-10T06:08:32.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.756+0000 7f32620d5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c105360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:32.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.756+0000 7f32620d5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c105360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57602/0 (socket says 172.21.15.89:57602) 2024-02-10T06:08:32.530 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.756+0000 7f32620d5700 1 -- 172.21.15.89:0/3816378374 learned_addr learned my addr 172.21.15.89:0/3816378374 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.756+0000 7f32620d5700 1 -- 172.21.15.89:0/3816378374 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c1059e0 con 0x7f325c076c40 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.756+0000 7f32620d5700 1 --2- 172.21.15.89:0/3816378374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c105360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3254009e50 tx=0x7f3254009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54548ce2a489fe12 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.756+0000 7f32610d3700 1 -- 172.21.15.89:0/3816378374 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32540092e0 con 0x7f325c076c40 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.756+0000 7f32610d3700 1 -- 172.21.15.89:0/3816378374 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3254012430 con 0x7f325c076c40 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.756+0000 7f32610d3700 1 -- 172.21.15.89:0/3816378374 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32540075a0 con 0x7f325c076c40 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.757+0000 7f3264339700 1 -- 172.21.15.89:0/3816378374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 msgr2=0x7f325c105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.757+0000 7f3264339700 1 --2- 172.21.15.89:0/3816378374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c105360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3254009e50 tx=0x7f3254009840 comp rx=0 tx=0).stop 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.757+0000 7f3264339700 1 -- 172.21.15.89:0/3816378374 shutdown_connections 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.757+0000 7f3264339700 1 --2- 172.21.15.89:0/3816378374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c105360 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.757+0000 7f3264339700 1 -- 172.21.15.89:0/3816378374 >> 172.21.15.89:0/3816378374 conn(0x7f325c0fcdd0 msgr2=0x7f325c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.757+0000 7f3264339700 1 -- 172.21.15.89:0/3816378374 shutdown_connections 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.757+0000 7f3264339700 1 -- 172.21.15.89:0/3816378374 wait complete. 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f3264339700 1 Processor -- start 2024-02-10T06:08:32.531 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f3264339700 1 -- start start 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f3264339700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c1115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f3264339700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c111b00 con 0x7f325c076c40 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f32620d5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c1115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f32620d5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c1115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57608/0 (socket says 172.21.15.89:57608) 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f32620d5700 1 -- 172.21.15.89:0/2895610682 learned_addr learned my addr 172.21.15.89:0/2895610682 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f32620d5700 1 -- 172.21.15.89:0/2895610682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3254009ae0 con 0x7f325c076c40 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f32620d5700 1 --2- 172.21.15.89:0/2895610682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c1115c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f32540127d0 tx=0x7f3254009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f3252ffd700 1 -- 172.21.15.89:0/2895610682 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32540092e0 con 0x7f325c076c40 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f3264339700 1 -- 172.21.15.89:0/2895610682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c111d00 con 0x7f325c076c40 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f3252ffd700 1 -- 172.21.15.89:0/2895610682 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3254007d30 con 0x7f325c076c40 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f3252ffd700 1 -- 172.21.15.89:0/2895610682 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32540259a0 con 0x7f325c076c40 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.758+0000 7f3264339700 1 -- 172.21.15.89:0/2895610682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325c1129a0 con 0x7f325c076c40 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.759+0000 7f3252ffd700 1 -- 172.21.15.89:0/2895610682 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f3254019050 con 0x7f325c076c40 2024-02-10T06:08:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.759+0000 7f3264339700 1 -- 172.21.15.89:0/2895610682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c05f5c0 con 0x7f325c076c40 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.759+0000 7f3252ffd700 1 --2- 172.21.15.89:0/2895610682 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f3248036790 0x7f3248038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.760+0000 7f3252ffd700 1 -- 172.21.15.89:0/2895610682 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f325404a1b0 con 0x7f325c076c40 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.760+0000 7f32618d4700 1 --2- 172.21.15.89:0/2895610682 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f3248036790 0x7f3248038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.760+0000 7f32618d4700 1 --2- 172.21.15.89:0/2895610682 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f3248036790 0x7f3248038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f324c009990 tx=0x7f324c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:30.764+0000 7f3252ffd700 1 -- 172.21.15.89:0/2895610682 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3254011570 con 0x7f325c076c40 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.001+0000 7f3264339700 1 -- 172.21.15.89:0/2895610682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f325c04bed0 con 0x7f325c076c40 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.932+0000 7f3252ffd700 1 -- 172.21.15.89:0/2895610682 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f325401c410 con 0x7f325c076c40 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.934+0000 7f3252ffd700 1 -- 172.21.15.89:0/2895610682 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f325402eba0 con 0x7f325c076c40 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.937+0000 7f3264339700 1 -- 172.21.15.89:0/2895610682 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f3248036790 msgr2=0x7f3248038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.937+0000 7f3264339700 1 --2- 172.21.15.89:0/2895610682 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f3248036790 0x7f3248038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f324c009990 tx=0x7f324c006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.937+0000 7f3264339700 1 -- 172.21.15.89:0/2895610682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 msgr2=0x7f325c1115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.937+0000 7f3264339700 1 --2- 172.21.15.89:0/2895610682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c1115c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f32540127d0 tx=0x7f3254009840 comp rx=0 tx=0).stop 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.937+0000 7f3264339700 1 -- 172.21.15.89:0/2895610682 shutdown_connections 2024-02-10T06:08:32.533 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.937+0000 7f3264339700 1 --2- 172.21.15.89:0/2895610682 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f3248036790 0x7f3248038c40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:32.534 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.937+0000 7f3264339700 1 --2- 172.21.15.89:0/2895610682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f325c076c40 0x7f325c1115c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:32.534 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.937+0000 7f3264339700 1 -- 172.21.15.89:0/2895610682 >> 172.21.15.89:0/2895610682 conn(0x7f325c0fcdd0 msgr2=0x7f325c0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:32.534 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.937+0000 7f3264339700 1 -- 172.21.15.89:0/2895610682 shutdown_connections 2024-02-10T06:08:32.534 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:31.937+0000 7f3264339700 1 -- 172.21.15.89:0/2895610682 wait complete. 2024-02-10T06:08:33.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:32 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2895610682' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T06:08:33.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:32 smithi089 ceph-mon[29217]: mgrmap e9: smithi089.pwmrye(active, since 23s) 2024-02-10T06:08:33.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:08:33.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-10T06:08:33.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T06:08:33.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "active_name": "smithi089.pwmrye", 2024-02-10T06:08:33.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T06:08:33.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:08:33.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.980+0000 7f57cab74700 1 Processor -- start 2024-02-10T06:08:33.815 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.980+0000 7f57cab74700 1 -- start start 2024-02-10T06:08:33.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.980+0000 7f57cab74700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:33.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.980+0000 7f57cab74700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c41030c0 con 0x7f57c4102770 2024-02-10T06:08:33.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.980+0000 7f57c8910700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:33.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.980+0000 7f57c8910700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57636/0 (socket says 172.21.15.89:57636) 2024-02-10T06:08:33.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.980+0000 7f57c8910700 1 -- 172.21.15.89:0/3873573551 learned_addr learned my addr 172.21.15.89:0/3873573551 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:33.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.981+0000 7f57c8910700 1 -- 172.21.15.89:0/3873573551 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c4103200 con 0x7f57c4102770 2024-02-10T06:08:33.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.981+0000 7f57c8910700 1 --2- 172.21.15.89:0/3873573551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4102b80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f57b80098e0 tx=0x7f57b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b18439a5faa22b server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:33.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.981+0000 7f57c37fe700 1 -- 172.21.15.89:0/3873573551 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b800b860 con 0x7f57c4102770 2024-02-10T06:08:33.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.981+0000 7f57c37fe700 1 -- 172.21.15.89:0/3873573551 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57b800be90 con 0x7f57c4102770 2024-02-10T06:08:33.816 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.981+0000 7f57c37fe700 1 -- 172.21.15.89:0/3873573551 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b80103c0 con 0x7f57c4102770 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.981+0000 7f57cab74700 1 -- 172.21.15.89:0/3873573551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 msgr2=0x7f57c4102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.981+0000 7f57cab74700 1 --2- 172.21.15.89:0/3873573551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4102b80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f57b80098e0 tx=0x7f57b8009250 comp rx=0 tx=0).stop 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.982+0000 7f57cab74700 1 -- 172.21.15.89:0/3873573551 shutdown_connections 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.982+0000 7f57cab74700 1 --2- 172.21.15.89:0/3873573551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4102b80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.982+0000 7f57cab74700 1 -- 172.21.15.89:0/3873573551 >> 172.21.15.89:0/3873573551 conn(0x7f57c40fded0 msgr2=0x7f57c4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.982+0000 7f57cab74700 1 -- 172.21.15.89:0/3873573551 shutdown_connections 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.982+0000 7f57cab74700 1 -- 172.21.15.89:0/3873573551 wait complete. 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.982+0000 7f57cab74700 1 Processor -- start 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.982+0000 7f57cab74700 1 -- start start 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.982+0000 7f57cab74700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.982+0000 7f57cab74700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4111a80 con 0x7f57c4102770 2024-02-10T06:08:33.817 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.983+0000 7f57c8910700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.983+0000 7f57c8910700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57648/0 (socket says 172.21.15.89:57648) 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.983+0000 7f57c8910700 1 -- 172.21.15.89:0/2205664745 learned_addr learned my addr 172.21.15.89:0/2205664745 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.983+0000 7f57c8910700 1 -- 172.21.15.89:0/2205664745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b80094f0 con 0x7f57c4102770 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.983+0000 7f57c8910700 1 --2- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4111540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f57b8011650 tx=0x7f57b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.983+0000 7f57c17fa700 1 -- 172.21.15.89:0/2205664745 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b8010890 con 0x7f57c4102770 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.983+0000 7f57c17fa700 1 -- 172.21.15.89:0/2205664745 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57b8010ec0 con 0x7f57c4102770 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.983+0000 7f57c17fa700 1 -- 172.21.15.89:0/2205664745 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b801a430 con 0x7f57c4102770 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.983+0000 7f57cab74700 1 -- 172.21.15.89:0/2205664745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c4111c80 con 0x7f57c4102770 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.983+0000 7f57cab74700 1 -- 172.21.15.89:0/2205664745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c4112920 con 0x7f57c4102770 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.984+0000 7f57c17fa700 1 -- 172.21.15.89:0/2205664745 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f57b801a660 con 0x7f57c4102770 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.984+0000 7f57cab74700 1 -- 172.21.15.89:0/2205664745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c405f5c0 con 0x7f57c4102770 2024-02-10T06:08:33.818 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.984+0000 7f57c17fa700 1 --2- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f57b4036830 0x7f57b4038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.984+0000 7f57c3fff700 1 -- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f57b4036830 msgr2=0x7f57b4038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1397975713 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.984+0000 7f57c17fa700 1 -- 172.21.15.89:0/2205664745 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f57b804a4e0 con 0x7f57c4102770 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.984+0000 7f57c3fff700 1 --2- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f57b4036830 0x7f57b4038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:32.989+0000 7f57c17fa700 1 -- 172.21.15.89:0/2205664745 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f57b8018db0 con 0x7f57c4102770 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.185+0000 7f57c3fff700 1 -- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f57b4036830 msgr2=0x7f57b4038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1397975713 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.185+0000 7f57c3fff700 1 --2- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f57b4036830 0x7f57b4038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.230+0000 7f57cab74700 1 -- 172.21.15.89:0/2205664745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f57c4112cc0 con 0x7f57c4102770 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.230+0000 7f57c17fa700 1 -- 172.21.15.89:0/2205664745 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f57b8018350 con 0x7f57c4102770 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.232+0000 7f57cab74700 1 -- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f57b4036830 msgr2=0x7f57b4038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.232+0000 7f57cab74700 1 --2- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f57b4036830 0x7f57b4038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.232+0000 7f57cab74700 1 -- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 msgr2=0x7f57c4111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.232+0000 7f57cab74700 1 --2- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4111540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f57b8011650 tx=0x7f57b8009250 comp rx=0 tx=0).stop 2024-02-10T06:08:33.819 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.233+0000 7f57cab74700 1 -- 172.21.15.89:0/2205664745 shutdown_connections 2024-02-10T06:08:33.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.233+0000 7f57cab74700 1 --2- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f57b4036830 0x7f57b4038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:33.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.233+0000 7f57cab74700 1 --2- 172.21.15.89:0/2205664745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57c4102770 0x7f57c4111540 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:33.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.233+0000 7f57cab74700 1 -- 172.21.15.89:0/2205664745 >> 172.21.15.89:0/2205664745 conn(0x7f57c40fded0 msgr2=0x7f57c40fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:33.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.233+0000 7f57cab74700 1 -- 172.21.15.89:0/2205664745 shutdown_connections 2024-02-10T06:08:33.820 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:33.233+0000 7f57cab74700 1 -- 172.21.15.89:0/2205664745 wait complete. 2024-02-10T06:08:33.820 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for the mgr to restart... 2024-02-10T06:08:33.820 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr epoch 9... 2024-02-10T06:08:34.061 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:33 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2205664745' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T06:08:39.271 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: Active manager daemon smithi089.pwmrye restarted 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: Activating manager daemon smithi089.pwmrye 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: osdmap e3: 0 total, 0 up, 0 in 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: mgrmap e10: smithi089.pwmrye(active, starting, since 0.00314563s) 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr metadata", "who": "smithi089.pwmrye", "id": "smithi089.pwmrye"}]: dispatch 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: Manager daemon smithi089.pwmrye is now available 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:39.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.pwmrye/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:08:39.273 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:38 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.pwmrye/trash_purge_schedule"}]: dispatch 2024-02-10T06:08:40.150 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-02-10T06:08:40.150 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.180+0000 7f53f4dba700 1 Processor -- start 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.180+0000 7f53f4dba700 1 -- start start 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.180+0000 7f53f4dba700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.180+0000 7f53f4dba700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f01030c0 con 0x7f53f0102770 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.180+0000 7f53ee59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.180+0000 7f53ee59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57664/0 (socket says 172.21.15.89:57664) 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.180+0000 7f53ee59c700 1 -- 172.21.15.89:0/1575160431 learned_addr learned my addr 172.21.15.89:0/1575160431 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.181+0000 7f53ee59c700 1 -- 172.21.15.89:0/1575160431 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f0103200 con 0x7f53f0102770 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.181+0000 7f53ee59c700 1 --2- 172.21.15.89:0/1575160431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0102b80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f53e40098e0 tx=0x7f53e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab611e823cb2148f server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.181+0000 7f53ed59a700 1 -- 172.21.15.89:0/1575160431 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e400b860 con 0x7f53f0102770 2024-02-10T06:08:40.151 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.181+0000 7f53ed59a700 1 -- 172.21.15.89:0/1575160431 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53e400be90 con 0x7f53f0102770 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.181+0000 7f53ed59a700 1 -- 172.21.15.89:0/1575160431 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e40103c0 con 0x7f53f0102770 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.182+0000 7f53f4dba700 1 -- 172.21.15.89:0/1575160431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 msgr2=0x7f53f0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.182+0000 7f53f4dba700 1 --2- 172.21.15.89:0/1575160431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0102b80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f53e40098e0 tx=0x7f53e4009250 comp rx=0 tx=0).stop 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.182+0000 7f53f4dba700 1 -- 172.21.15.89:0/1575160431 shutdown_connections 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.182+0000 7f53f4dba700 1 --2- 172.21.15.89:0/1575160431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0102b80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.182+0000 7f53f4dba700 1 -- 172.21.15.89:0/1575160431 >> 172.21.15.89:0/1575160431 conn(0x7f53f00fded0 msgr2=0x7f53f0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.182+0000 7f53f4dba700 1 -- 172.21.15.89:0/1575160431 shutdown_connections 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.182+0000 7f53f4dba700 1 -- 172.21.15.89:0/1575160431 wait complete. 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.183+0000 7f53f4dba700 1 Processor -- start 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.183+0000 7f53f4dba700 1 -- start start 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.183+0000 7f53f4dba700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.183+0000 7f53f4dba700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f0111a80 con 0x7f53f0102770 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.183+0000 7f53ee59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:40.152 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.183+0000 7f53ee59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57672/0 (socket says 172.21.15.89:57672) 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.183+0000 7f53ee59c700 1 -- 172.21.15.89:0/1423704051 learned_addr learned my addr 172.21.15.89:0/1423704051 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.183+0000 7f53ee59c700 1 -- 172.21.15.89:0/1423704051 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e40094f0 con 0x7f53f0102770 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.183+0000 7f53ee59c700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0111540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f53e4011650 tx=0x7f53e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.183+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e4010890 con 0x7f53f0102770 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.184+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53e4010ec0 con 0x7f53f0102770 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.184+0000 7f53f4dba700 1 -- 172.21.15.89:0/1423704051 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f0111c80 con 0x7f53f0102770 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.184+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e401a430 con 0x7f53f0102770 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.184+0000 7f53f4dba700 1 -- 172.21.15.89:0/1423704051 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f0112920 con 0x7f53f0102770 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.185+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f53e40247a0 con 0x7f53f0102770 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.185+0000 7f53deffd700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 0x7f53d8038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.185+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 --> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53d80393f0 con 0x7f53d8036830 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.185+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f53e404a580 con 0x7f53f0102770 2024-02-10T06:08:40.153 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.185+0000 7f53edd9b700 1 -- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 msgr2=0x7f53d8038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1397975713 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.185+0000 7f53edd9b700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 0x7f53d8038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.386+0000 7f53edd9b700 1 -- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 msgr2=0x7f53d8038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1397975713 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.386+0000 7f53edd9b700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 0x7f53d8038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.787+0000 7f53edd9b700 1 -- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 msgr2=0x7f53d8038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1397975713 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:34.787+0000 7f53edd9b700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 0x7f53d8038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:35.588+0000 7f53edd9b700 1 -- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 msgr2=0x7f53d8038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1397975713 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:35.588+0000 7f53edd9b700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 0x7f53d8038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:37.189+0000 7f53edd9b700 1 -- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 msgr2=0x7f53d8038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1397975713 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:37.189+0000 7f53edd9b700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 0x7f53d8038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:38.559+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7f53e402d3f0 con 0x7f53f0102770 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:38.559+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 msgr2=0x7f53d8038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:38.559+0000 7f53deffd700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 0x7f53d8038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.560+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f53e4018c50 con 0x7f53f0102770 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.560+0000 7f53deffd700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f53d803a070 0x7f53d803c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:40.154 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.560+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 --> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53d80393f0 con 0x7f53d803a070 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.561+0000 7f53edd9b700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f53d803a070 0x7f53d803c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.561+0000 7f53edd9b700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f53d803a070 0x7f53d803c450 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f53e00039d0 tx=0x7f53e0009350 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.562+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 <== mgr.14162 v2:172.21.15.89:6800/1955086590 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f53d80393f0 con 0x7f53d803a070 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.567+0000 7f53f4dba700 1 -- 172.21.15.89:0/1423704051 --> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f53f0112110 con 0x7f53d803a070 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53deffd700 1 -- 172.21.15.89:0/1423704051 <== mgr.14162 v2:172.21.15.89:6800/1955086590 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f53f0112110 con 0x7f53d803a070 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53f4dba700 1 -- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f53d803a070 msgr2=0x7f53d803c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53f4dba700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f53d803a070 0x7f53d803c450 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f53e00039d0 tx=0x7f53e0009350 comp rx=0 tx=0).stop 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53f4dba700 1 -- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 msgr2=0x7f53f0111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53f4dba700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0111540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f53e4011650 tx=0x7f53e4009250 comp rx=0 tx=0).stop 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53f4dba700 1 -- 172.21.15.89:0/1423704051 shutdown_connections 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53f4dba700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f53d803a070 0x7f53d803c450 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53f4dba700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102770 0x7f53f0111540 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:40.155 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53f4dba700 1 --2- 172.21.15.89:0/1423704051 >> [v2:172.21.15.89:6800/1397975713,v1:172.21.15.89:6801/1397975713] conn(0x7f53d8036830 0x7f53d8038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:40.156 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53f4dba700 1 -- 172.21.15.89:0/1423704051 >> 172.21.15.89:0/1423704051 conn(0x7f53f00fded0 msgr2=0x7f53f00fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:40.156 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53f4dba700 1 -- 172.21.15.89:0/1423704051 shutdown_connections 2024-02-10T06:08:40.156 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:39.568+0000 7f53f4dba700 1 -- 172.21.15.89:0/1423704051 wait complete. 2024-02-10T06:08:40.156 INFO:teuthology.orchestra.run.smithi089.stdout:mgr epoch 9 is available 2024-02-10T06:08:40.156 INFO:teuthology.orchestra.run.smithi089.stdout:Generating a dashboard self-signed certificate... 2024-02-10T06:08:40.267 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:39 smithi089 ceph-mon[29217]: [10/Feb/2024:06:08:38] ENGINE Bus STARTING 2024-02-10T06:08:40.267 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:39 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:40.267 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:39 smithi089 ceph-mon[29217]: mgrmap e11: smithi089.pwmrye(active, since 1.00534s) 2024-02-10T06:08:41.270 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:40 smithi089 ceph-mon[29217]: [10/Feb/2024:06:08:39] ENGINE Serving on https://172.21.15.89:7150 2024-02-10T06:08:41.270 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:40 smithi089 ceph-mon[29217]: [10/Feb/2024:06:08:39] ENGINE Bus STARTED 2024-02-10T06:08:41.270 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:40 smithi089 ceph-mon[29217]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T06:08:41.270 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:40 smithi089 ceph-mon[29217]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T06:08:41.270 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:40 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:41.270 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:40 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:41.270 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:40 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:41.406 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-10T06:08:41.406 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.566+0000 7fb14f06c700 1 Processor -- start 2024-02-10T06:08:41.406 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.566+0000 7fb14f06c700 1 -- start start 2024-02-10T06:08:41.406 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.566+0000 7fb14f06c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb148101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:41.406 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.566+0000 7fb14f06c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb148101fc0 con 0x7fb148101670 2024-02-10T06:08:41.406 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.566+0000 7fb14ce08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb148101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:41.406 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.566+0000 7fb14ce08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb148101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45902/0 (socket says 172.21.15.89:45902) 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.566+0000 7fb14ce08700 1 -- 172.21.15.89:0/1916857706 learned_addr learned my addr 172.21.15.89:0/1916857706 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.566+0000 7fb14ce08700 1 -- 172.21.15.89:0/1916857706 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb148102100 con 0x7fb148101670 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.566+0000 7fb14ce08700 1 --2- 172.21.15.89:0/1916857706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb148101a80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00bb80 tx=0x7fb13c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54feec11047e4db2 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.567+0000 7fb1477fe700 1 -- 172.21.15.89:0/1916857706 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb13c00dd00 con 0x7fb148101670 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.567+0000 7fb1477fe700 1 -- 172.21.15.89:0/1916857706 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb13c00de60 con 0x7fb148101670 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.567+0000 7fb1477fe700 1 -- 172.21.15.89:0/1916857706 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb13c018920 con 0x7fb148101670 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.567+0000 7fb14f06c700 1 -- 172.21.15.89:0/1916857706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 msgr2=0x7fb148101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.567+0000 7fb14f06c700 1 --2- 172.21.15.89:0/1916857706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb148101a80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00bb80 tx=0x7fb13c00b4f0 comp rx=0 tx=0).stop 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.567+0000 7fb14f06c700 1 -- 172.21.15.89:0/1916857706 shutdown_connections 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.567+0000 7fb14f06c700 1 --2- 172.21.15.89:0/1916857706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb148101a80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.567+0000 7fb14f06c700 1 -- 172.21.15.89:0/1916857706 >> 172.21.15.89:0/1916857706 conn(0x7fb1480fcdd0 msgr2=0x7fb1480ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.568+0000 7fb14f06c700 1 -- 172.21.15.89:0/1916857706 shutdown_connections 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.568+0000 7fb14f06c700 1 -- 172.21.15.89:0/1916857706 wait complete. 2024-02-10T06:08:41.407 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.568+0000 7fb14f06c700 1 Processor -- start 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.568+0000 7fb14f06c700 1 -- start start 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.568+0000 7fb14f06c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb14810f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.568+0000 7fb14f06c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14810f880 con 0x7fb148101670 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.569+0000 7fb14ce08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb14810f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.569+0000 7fb14ce08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb14810f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45918/0 (socket says 172.21.15.89:45918) 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.569+0000 7fb14ce08700 1 -- 172.21.15.89:0/547447328 learned_addr learned my addr 172.21.15.89:0/547447328 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.569+0000 7fb14ce08700 1 -- 172.21.15.89:0/547447328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c00b790 con 0x7fb148101670 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.569+0000 7fb14ce08700 1 --2- 172.21.15.89:0/547447328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb14810f340 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0106a0 tx=0x7fb13c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.569+0000 7fb1457fa700 1 -- 172.21.15.89:0/547447328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb13c010bf0 con 0x7fb148101670 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.569+0000 7fb14f06c700 1 -- 172.21.15.89:0/547447328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14810fa80 con 0x7fb148101670 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.569+0000 7fb1457fa700 1 -- 172.21.15.89:0/547447328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb13c01d050 con 0x7fb148101670 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.569+0000 7fb1457fa700 1 -- 172.21.15.89:0/547447328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb13c022620 con 0x7fb148101670 2024-02-10T06:08:41.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.569+0000 7fb14f06c700 1 -- 172.21.15.89:0/547447328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb148110720 con 0x7fb148101670 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.570+0000 7fb1457fa700 1 -- 172.21.15.89:0/547447328 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fb13c022780 con 0x7fb148101670 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.570+0000 7fb14f06c700 1 -- 172.21.15.89:0/547447328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14805f5c0 con 0x7fb148101670 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.570+0000 7fb1457fa700 1 --2- 172.21.15.89:0/547447328 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb138036710 0x7fb138038bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.570+0000 7fb1457fa700 1 -- 172.21.15.89:0/547447328 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb13c014070 con 0x7fb148101670 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.570+0000 7fb147fff700 1 --2- 172.21.15.89:0/547447328 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb138036710 0x7fb138038bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.571+0000 7fb147fff700 1 --2- 172.21.15.89:0/547447328 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb138036710 0x7fb138038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb134009990 tx=0x7fb134006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.575+0000 7fb1457fa700 1 -- 172.21.15.89:0/547447328 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb13c051030 con 0x7fb148101670 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.772+0000 7fb14f06c700 1 -- 172.21.15.89:0/547447328 --> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1481112a0 con 0x7fb138036710 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.882+0000 7fb1457fa700 1 -- 172.21.15.89:0/547447328 <== mgr.14162 v2:172.21.15.89:6800/1955086590 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fb1481112a0 con 0x7fb138036710 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.884+0000 7fb14f06c700 1 -- 172.21.15.89:0/547447328 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb138036710 msgr2=0x7fb138038bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.884+0000 7fb14f06c700 1 --2- 172.21.15.89:0/547447328 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb138036710 0x7fb138038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb134009990 tx=0x7fb134006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.884+0000 7fb14f06c700 1 -- 172.21.15.89:0/547447328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 msgr2=0x7fb14810f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:41.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.884+0000 7fb14f06c700 1 --2- 172.21.15.89:0/547447328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb14810f340 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0106a0 tx=0x7fb13c00b4f0 comp rx=0 tx=0).stop 2024-02-10T06:08:41.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.885+0000 7fb14f06c700 1 -- 172.21.15.89:0/547447328 shutdown_connections 2024-02-10T06:08:41.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.885+0000 7fb14f06c700 1 --2- 172.21.15.89:0/547447328 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb138036710 0x7fb138038bc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:41.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.885+0000 7fb14f06c700 1 --2- 172.21.15.89:0/547447328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb148101670 0x7fb14810f340 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:41.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.885+0000 7fb14f06c700 1 -- 172.21.15.89:0/547447328 >> 172.21.15.89:0/547447328 conn(0x7fb1480fcdd0 msgr2=0x7fb1480ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:41.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.885+0000 7fb14f06c700 1 -- 172.21.15.89:0/547447328 shutdown_connections 2024-02-10T06:08:41.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:40.885+0000 7fb14f06c700 1 -- 172.21.15.89:0/547447328 wait complete. 2024-02-10T06:08:41.410 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial admin user... 2024-02-10T06:08:42.058 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:41 smithi089 ceph-mon[29217]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:42.058 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:41 smithi089 ceph-mon[29217]: mgrmap e12: smithi089.pwmrye(active, since 2s) 2024-02-10T06:08:42.058 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:41 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:42.839 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$x29/J.aCH954z.UL6oI.T.Uo8MtLOi1m2l9ODDZd3Ehg9idAyV2cC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707545322, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T06:08:42.840 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.828+0000 7f831a414700 1 Processor -- start 2024-02-10T06:08:42.840 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.828+0000 7f831a414700 1 -- start start 2024-02-10T06:08:42.840 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.828+0000 7f831a414700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f831410cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:42.840 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.828+0000 7f831a414700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831406e680 con 0x7f831406e270 2024-02-10T06:08:42.840 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.828+0000 7f8313fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f831410cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:42.840 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.828+0000 7f8313fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f831410cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45934/0 (socket says 172.21.15.89:45934) 2024-02-10T06:08:42.840 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.828+0000 7f8313fff700 1 -- 172.21.15.89:0/3476040042 learned_addr learned my addr 172.21.15.89:0/3476040042 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:42.840 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.828+0000 7f8313fff700 1 -- 172.21.15.89:0/3476040042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831410d280 con 0x7f831406e270 2024-02-10T06:08:42.840 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.828+0000 7f8313fff700 1 --2- 172.21.15.89:0/3476040042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f831410cd40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0098e0 tx=0x7f82fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52490b5529eb2092 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:42.840 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.828+0000 7f8312ffd700 1 -- 172.21.15.89:0/3476040042 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82fc00b860 con 0x7f831406e270 2024-02-10T06:08:42.840 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.828+0000 7f8312ffd700 1 -- 172.21.15.89:0/3476040042 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82fc00be90 con 0x7f831406e270 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.829+0000 7f8312ffd700 1 -- 172.21.15.89:0/3476040042 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82fc0103c0 con 0x7f831406e270 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.829+0000 7f831a414700 1 -- 172.21.15.89:0/3476040042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 msgr2=0x7f831410cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.829+0000 7f831a414700 1 --2- 172.21.15.89:0/3476040042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f831410cd40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0098e0 tx=0x7f82fc009250 comp rx=0 tx=0).stop 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.829+0000 7f831a414700 1 -- 172.21.15.89:0/3476040042 shutdown_connections 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.829+0000 7f831a414700 1 --2- 172.21.15.89:0/3476040042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f831410cd40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.829+0000 7f831a414700 1 -- 172.21.15.89:0/3476040042 >> 172.21.15.89:0/3476040042 conn(0x7f8314069d90 msgr2=0x7f831406c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.829+0000 7f831a414700 1 -- 172.21.15.89:0/3476040042 shutdown_connections 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.829+0000 7f831a414700 1 -- 172.21.15.89:0/3476040042 wait complete. 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.830+0000 7f831a414700 1 Processor -- start 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.830+0000 7f831a414700 1 -- start start 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.830+0000 7f831a414700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f83141102f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.830+0000 7f831a414700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831410e770 con 0x7f831406e270 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.830+0000 7f8313fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f83141102f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:42.841 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.830+0000 7f8313fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f83141102f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45940/0 (socket says 172.21.15.89:45940) 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.830+0000 7f8313fff700 1 -- 172.21.15.89:0/686705840 learned_addr learned my addr 172.21.15.89:0/686705840 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.830+0000 7f8313fff700 1 -- 172.21.15.89:0/686705840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc0094f0 con 0x7f831406e270 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.830+0000 7f8313fff700 1 --2- 172.21.15.89:0/686705840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f83141102f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00b440 tx=0x7f82fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.831+0000 7f8310ff9700 1 -- 172.21.15.89:0/686705840 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82fc010790 con 0x7f831406e270 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.831+0000 7f831a414700 1 -- 172.21.15.89:0/686705840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831410e970 con 0x7f831406e270 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.831+0000 7f8310ff9700 1 -- 172.21.15.89:0/686705840 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82fc010dc0 con 0x7f831406e270 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.831+0000 7f8310ff9700 1 -- 172.21.15.89:0/686705840 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82fc024e50 con 0x7f831406e270 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.831+0000 7f831a414700 1 -- 172.21.15.89:0/686705840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831410ee00 con 0x7f831406e270 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.831+0000 7f8310ff9700 1 -- 172.21.15.89:0/686705840 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f82fc01a510 con 0x7f831406e270 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.832+0000 7f831a414700 1 -- 172.21.15.89:0/686705840 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8314113fd0 con 0x7f831406e270 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.832+0000 7f8310ff9700 1 --2- 172.21.15.89:0/686705840 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f8300036790 0x7f8300038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.832+0000 7f8310ff9700 1 -- 172.21.15.89:0/686705840 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f82fc04a630 con 0x7f831406e270 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.832+0000 7f83137fe700 1 --2- 172.21.15.89:0/686705840 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f8300036790 0x7f8300038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.832+0000 7f83137fe700 1 --2- 172.21.15.89:0/686705840 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f8300036790 0x7f8300038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8304009990 tx=0x7f8304006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:41.836+0000 7f8310ff9700 1 -- 172.21.15.89:0/686705840 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f82fc018db0 con 0x7f831406e270 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.057+0000 7f831a414700 1 -- 172.21.15.89:0/686705840 --> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f831405f3e0 con 0x7f8300036790 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.283+0000 7f8310ff9700 1 -- 172.21.15.89:0/686705840 <== mgr.14162 v2:172.21.15.89:6800/1955086590 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f831405f3e0 con 0x7f8300036790 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.285+0000 7f831a414700 1 -- 172.21.15.89:0/686705840 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f8300036790 msgr2=0x7f8300038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.285+0000 7f831a414700 1 --2- 172.21.15.89:0/686705840 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f8300036790 0x7f8300038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8304009990 tx=0x7f8304006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.285+0000 7f831a414700 1 -- 172.21.15.89:0/686705840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 msgr2=0x7f83141102f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.285+0000 7f831a414700 1 --2- 172.21.15.89:0/686705840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f83141102f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00b440 tx=0x7f82fc009250 comp rx=0 tx=0).stop 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.285+0000 7f831a414700 1 -- 172.21.15.89:0/686705840 shutdown_connections 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.285+0000 7f831a414700 1 --2- 172.21.15.89:0/686705840 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f8300036790 0x7f8300038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.285+0000 7f831a414700 1 --2- 172.21.15.89:0/686705840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f831406e270 0x7f83141102f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.285+0000 7f831a414700 1 -- 172.21.15.89:0/686705840 >> 172.21.15.89:0/686705840 conn(0x7f8314069d90 msgr2=0x7f831406b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.285+0000 7f831a414700 1 -- 172.21.15.89:0/686705840 shutdown_connections 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:42.285+0000 7f831a414700 1 -- 172.21.15.89:0/686705840 wait complete. 2024-02-10T06:08:42.843 INFO:teuthology.orchestra.run.smithi089.stdout:Fetching dashboard port number... 2024-02-10T06:08:43.386 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:43 smithi089 ceph-mon[29217]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:43.386 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:43 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 8443 2024-02-10T06:08:44.123 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.215+0000 7f345b65d700 1 Processor -- start 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.215+0000 7f345b65d700 1 -- start start 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.215+0000 7f345b65d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f34540760e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.216+0000 7f345b65d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3454077fe0 con 0x7f3454077bd0 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.216+0000 7f34593f9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f34540760e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.216+0000 7f34593f9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f34540760e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45944/0 (socket says 172.21.15.89:45944) 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.216+0000 7f34593f9700 1 -- 172.21.15.89:0/1242421662 learned_addr learned my addr 172.21.15.89:0/1242421662 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.216+0000 7f34593f9700 1 -- 172.21.15.89:0/1242421662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3454076620 con 0x7f3454077bd0 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.216+0000 7f34593f9700 1 --2- 172.21.15.89:0/1242421662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f34540760e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f344c009e50 tx=0x7f344c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905b720c05cd7b16 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.216+0000 7f344bfff700 1 -- 172.21.15.89:0/1242421662 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f344c0092e0 con 0x7f3454077bd0 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.216+0000 7f344bfff700 1 -- 172.21.15.89:0/1242421662 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f344c012430 con 0x7f3454077bd0 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.216+0000 7f344bfff700 1 -- 172.21.15.89:0/1242421662 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f344c0075a0 con 0x7f3454077bd0 2024-02-10T06:08:44.124 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.217+0000 7f345b65d700 1 -- 172.21.15.89:0/1242421662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 msgr2=0x7f34540760e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.217+0000 7f345b65d700 1 --2- 172.21.15.89:0/1242421662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f34540760e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f344c009e50 tx=0x7f344c009840 comp rx=0 tx=0).stop 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.217+0000 7f345b65d700 1 -- 172.21.15.89:0/1242421662 shutdown_connections 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.217+0000 7f345b65d700 1 --2- 172.21.15.89:0/1242421662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f34540760e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.217+0000 7f345b65d700 1 -- 172.21.15.89:0/1242421662 >> 172.21.15.89:0/1242421662 conn(0x7f34540fded0 msgr2=0x7f3454100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.217+0000 7f345b65d700 1 -- 172.21.15.89:0/1242421662 shutdown_connections 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.217+0000 7f345b65d700 1 -- 172.21.15.89:0/1242421662 wait complete. 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.217+0000 7f345b65d700 1 Processor -- start 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f345b65d700 1 -- start start 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f345b65d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f3454113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f345b65d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3454113d60 con 0x7f3454077bd0 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f34593f9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f3454113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f34593f9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f3454113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45946/0 (socket says 172.21.15.89:45946) 2024-02-10T06:08:44.125 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f34593f9700 1 -- 172.21.15.89:0/1468807388 learned_addr learned my addr 172.21.15.89:0/1468807388 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f34593f9700 1 -- 172.21.15.89:0/1468807388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c009ae0 con 0x7f3454077bd0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f34593f9700 1 --2- 172.21.15.89:0/1468807388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f3454113820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f344c012de0 tx=0x7f344c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f3449ffb700 1 -- 172.21.15.89:0/1468807388 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f344c0092e0 con 0x7f3454077bd0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f3449ffb700 1 -- 172.21.15.89:0/1468807388 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f344c007d30 con 0x7f3454077bd0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f345b65d700 1 -- 172.21.15.89:0/1468807388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3454113f60 con 0x7f3454077bd0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.218+0000 7f3449ffb700 1 -- 172.21.15.89:0/1468807388 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f344c025a00 con 0x7f3454077bd0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.219+0000 7f345b65d700 1 -- 172.21.15.89:0/1468807388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3454114c00 con 0x7f3454077bd0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.219+0000 7f3449ffb700 1 -- 172.21.15.89:0/1468807388 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f344c019050 con 0x7f3454077bd0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.219+0000 7f345b65d700 1 -- 172.21.15.89:0/1468807388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345405f5c0 con 0x7f3454077bd0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.220+0000 7f3449ffb700 1 --2- 172.21.15.89:0/1468807388 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3440036830 0x7f3440038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.220+0000 7f3449ffb700 1 -- 172.21.15.89:0/1468807388 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f344c04a090 con 0x7f3454077bd0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.220+0000 7f3458bf8700 1 --2- 172.21.15.89:0/1468807388 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3440036830 0x7f3440038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.220+0000 7f3458bf8700 1 --2- 172.21.15.89:0/1468807388 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3440036830 0x7f3440038ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3444009990 tx=0x7f3444006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:44.126 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.224+0000 7f3449ffb700 1 -- 172.21.15.89:0/1468807388 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f344c07bbd0 con 0x7f3454077bd0 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.422+0000 7f345b65d700 1 -- 172.21.15.89:0/1468807388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f345404bed0 con 0x7f3454077bd0 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.422+0000 7f3449ffb700 1 -- 172.21.15.89:0/1468807388 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f344c017070 con 0x7f3454077bd0 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.424+0000 7f345b65d700 1 -- 172.21.15.89:0/1468807388 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3440036830 msgr2=0x7f3440038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.424+0000 7f345b65d700 1 --2- 172.21.15.89:0/1468807388 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3440036830 0x7f3440038ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3444009990 tx=0x7f3444006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.424+0000 7f345b65d700 1 -- 172.21.15.89:0/1468807388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 msgr2=0x7f3454113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.424+0000 7f345b65d700 1 --2- 172.21.15.89:0/1468807388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f3454113820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f344c012de0 tx=0x7f344c009840 comp rx=0 tx=0).stop 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.424+0000 7f345b65d700 1 -- 172.21.15.89:0/1468807388 shutdown_connections 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.424+0000 7f345b65d700 1 --2- 172.21.15.89:0/1468807388 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3440036830 0x7f3440038ce0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.424+0000 7f345b65d700 1 --2- 172.21.15.89:0/1468807388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3454077bd0 0x7f3454113820 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.424+0000 7f345b65d700 1 -- 172.21.15.89:0/1468807388 >> 172.21.15.89:0/1468807388 conn(0x7f34540fded0 msgr2=0x7f34540ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.424+0000 7f345b65d700 1 -- 172.21.15.89:0/1468807388 shutdown_connections 2024-02-10T06:08:44.127 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:43.424+0000 7f345b65d700 1 -- 172.21.15.89:0/1468807388 wait complete. 2024-02-10T06:08:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T06:08:44.130 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-02-10T06:08:44.137 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T06:08:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-02-10T06:08:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-02-10T06:08:44.138 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T06:08:44.139 INFO:teuthology.orchestra.run.smithi089.stdout:Ceph Dashboard is now available at: 2024-02-10T06:08:44.139 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:08:44.139 INFO:teuthology.orchestra.run.smithi089.stdout: URL: https://smithi089.front.sepia.ceph.com:8443/ 2024-02-10T06:08:44.139 INFO:teuthology.orchestra.run.smithi089.stdout: User: admin 2024-02-10T06:08:44.140 INFO:teuthology.orchestra.run.smithi089.stdout: Password: onjrpcg8ji 2024-02-10T06:08:44.140 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:08:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:Saving cluster configuration to /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config directory 2024-02-10T06:08:44.140 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling autotune for osd_memory_target 2024-02-10T06:08:44.527 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:44 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1468807388' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T06:08:45.732 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.953+0000 7fb4089ca700 1 Processor -- start 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.953+0000 7fb4089ca700 1 -- start start 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.953+0000 7fb4089ca700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb404101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.953+0000 7fb4089ca700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb404101fc0 con 0x7fb404101670 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.953+0000 7fb40259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb404101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.953+0000 7fb40259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb404101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45950/0 (socket says 172.21.15.89:45950) 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.953+0000 7fb40259c700 1 -- 172.21.15.89:0/3739599483 learned_addr learned my addr 172.21.15.89:0/3739599483 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.954+0000 7fb40259c700 1 -- 172.21.15.89:0/3739599483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb404102100 con 0x7fb404101670 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.954+0000 7fb40259c700 1 --2- 172.21.15.89:0/3739599483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb404101a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40098e0 tx=0x7fb3f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b5b4a0a1f7b32f server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.954+0000 7fb401d9b700 1 -- 172.21.15.89:0/3739599483 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f400b860 con 0x7fb404101670 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.954+0000 7fb401d9b700 1 -- 172.21.15.89:0/3739599483 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3f400be90 con 0x7fb404101670 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.954+0000 7fb401d9b700 1 -- 172.21.15.89:0/3739599483 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f40103c0 con 0x7fb404101670 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.955+0000 7fb4089ca700 1 -- 172.21.15.89:0/3739599483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 msgr2=0x7fb404101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:45.733 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.955+0000 7fb4089ca700 1 --2- 172.21.15.89:0/3739599483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb404101a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40098e0 tx=0x7fb3f4009250 comp rx=0 tx=0).stop 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.955+0000 7fb4089ca700 1 -- 172.21.15.89:0/3739599483 shutdown_connections 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.955+0000 7fb4089ca700 1 --2- 172.21.15.89:0/3739599483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb404101a80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.955+0000 7fb4089ca700 1 -- 172.21.15.89:0/3739599483 >> 172.21.15.89:0/3739599483 conn(0x7fb4040fcdd0 msgr2=0x7fb4040ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.955+0000 7fb4089ca700 1 -- 172.21.15.89:0/3739599483 shutdown_connections 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.955+0000 7fb4089ca700 1 -- 172.21.15.89:0/3739599483 wait complete. 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.955+0000 7fb4089ca700 1 Processor -- start 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.955+0000 7fb4089ca700 1 -- start start 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.955+0000 7fb4089ca700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb40410f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.956+0000 7fb4089ca700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40410f8e0 con 0x7fb404101670 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.956+0000 7fb40259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb40410f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.956+0000 7fb40259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb40410f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45964/0 (socket says 172.21.15.89:45964) 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.956+0000 7fb40259c700 1 -- 172.21.15.89:0/306895884 learned_addr learned my addr 172.21.15.89:0/306895884 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.956+0000 7fb40259c700 1 -- 172.21.15.89:0/306895884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f40094f0 con 0x7fb404101670 2024-02-10T06:08:45.734 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.956+0000 7fb40259c700 1 --2- 172.21.15.89:0/306895884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb40410f3a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40090b0 tx=0x7fb3f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.956+0000 7fb3fb7fe700 1 -- 172.21.15.89:0/306895884 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f4010840 con 0x7fb404101670 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.956+0000 7fb3fb7fe700 1 -- 172.21.15.89:0/306895884 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3f4010e70 con 0x7fb404101670 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.956+0000 7fb4089ca700 1 -- 172.21.15.89:0/306895884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40410fae0 con 0x7fb404101670 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.956+0000 7fb3fb7fe700 1 -- 172.21.15.89:0/306895884 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f4024e40 con 0x7fb404101670 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.956+0000 7fb4089ca700 1 -- 172.21.15.89:0/306895884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb404110780 con 0x7fb404101670 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.957+0000 7fb3fb7fe700 1 -- 172.21.15.89:0/306895884 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fb3f401a4c0 con 0x7fb404101670 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.957+0000 7fb4089ca700 1 -- 172.21.15.89:0/306895884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40405f5c0 con 0x7fb404101670 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.957+0000 7fb3fb7fe700 1 --2- 172.21.15.89:0/306895884 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb3f00389c0 0x7fb3f003ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.957+0000 7fb3fb7fe700 1 -- 172.21.15.89:0/306895884 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb3f404ab00 con 0x7fb404101670 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.958+0000 7fb3f9d9b700 1 --2- 172.21.15.89:0/306895884 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb3f00389c0 0x7fb3f003ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.958+0000 7fb3f9d9b700 1 --2- 172.21.15.89:0/306895884 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb3f00389c0 0x7fb3f003ae70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009990 tx=0x7fb3ec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:44.962+0000 7fb3fb7fe700 1 -- 172.21.15.89:0/306895884 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb3f4018940 con 0x7fb404101670 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.144+0000 7fb4089ca700 1 -- 172.21.15.89:0/306895884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fb40404bed0 con 0x7fb404101670 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.144+0000 7fb3fb7fe700 1 -- 172.21.15.89:0/306895884 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fb3f4018350 con 0x7fb404101670 2024-02-10T06:08:45.735 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.147+0000 7fb4089ca700 1 -- 172.21.15.89:0/306895884 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb3f00389c0 msgr2=0x7fb3f003ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:45.736 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.147+0000 7fb4089ca700 1 --2- 172.21.15.89:0/306895884 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb3f00389c0 0x7fb3f003ae70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009990 tx=0x7fb3ec006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:45.736 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.147+0000 7fb4089ca700 1 -- 172.21.15.89:0/306895884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 msgr2=0x7fb40410f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:45.736 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.147+0000 7fb4089ca700 1 --2- 172.21.15.89:0/306895884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb40410f3a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40090b0 tx=0x7fb3f4009250 comp rx=0 tx=0).stop 2024-02-10T06:08:45.736 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.147+0000 7fb4089ca700 1 -- 172.21.15.89:0/306895884 shutdown_connections 2024-02-10T06:08:45.736 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.147+0000 7fb4089ca700 1 --2- 172.21.15.89:0/306895884 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fb3f00389c0 0x7fb3f003ae70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:45.736 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.147+0000 7fb4089ca700 1 --2- 172.21.15.89:0/306895884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb404101670 0x7fb40410f3a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:45.736 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.147+0000 7fb4089ca700 1 -- 172.21.15.89:0/306895884 >> 172.21.15.89:0/306895884 conn(0x7fb4040fcdd0 msgr2=0x7fb4041063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:45.736 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.147+0000 7fb4089ca700 1 -- 172.21.15.89:0/306895884 shutdown_connections 2024-02-10T06:08:45.736 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:45.147+0000 7fb4089ca700 1 -- 172.21.15.89:0/306895884 wait complete. 2024-02-10T06:08:46.436 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:46 smithi089 ceph-mon[29217]: mgrmap e13: smithi089.pwmrye(active, since 6s) 2024-02-10T06:08:46.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.140+0000 7f0a7a92f700 1 Processor -- start 2024-02-10T06:08:46.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.140+0000 7f0a7a92f700 1 -- start start 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.141+0000 7f0a7a92f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.141+0000 7f0a7a92f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a741030c0 con 0x7f0a74102770 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.141+0000 7f0a73fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.141+0000 7f0a73fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45976/0 (socket says 172.21.15.89:45976) 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.141+0000 7f0a73fff700 1 -- 172.21.15.89:0/2305943042 learned_addr learned my addr 172.21.15.89:0/2305943042 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.141+0000 7f0a73fff700 1 -- 172.21.15.89:0/2305943042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a74103200 con 0x7f0a74102770 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.141+0000 7f0a73fff700 1 --2- 172.21.15.89:0/2305943042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74102b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0a680098e0 tx=0x7f0a68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48607b894b1c2170 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.141+0000 7f0a72ffd700 1 -- 172.21.15.89:0/2305943042 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a6800b860 con 0x7f0a74102770 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.141+0000 7f0a72ffd700 1 -- 172.21.15.89:0/2305943042 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a6800be90 con 0x7f0a74102770 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.141+0000 7f0a72ffd700 1 -- 172.21.15.89:0/2305943042 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a680103c0 con 0x7f0a74102770 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.142+0000 7f0a7a92f700 1 -- 172.21.15.89:0/2305943042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 msgr2=0x7f0a74102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.142+0000 7f0a7a92f700 1 --2- 172.21.15.89:0/2305943042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74102b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0a680098e0 tx=0x7f0a68009250 comp rx=0 tx=0).stop 2024-02-10T06:08:46.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.142+0000 7f0a7a92f700 1 -- 172.21.15.89:0/2305943042 shutdown_connections 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.142+0000 7f0a7a92f700 1 --2- 172.21.15.89:0/2305943042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74102b80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.142+0000 7f0a7a92f700 1 -- 172.21.15.89:0/2305943042 >> 172.21.15.89:0/2305943042 conn(0x7f0a740fded0 msgr2=0x7f0a74100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.142+0000 7f0a7a92f700 1 -- 172.21.15.89:0/2305943042 shutdown_connections 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.142+0000 7f0a7a92f700 1 -- 172.21.15.89:0/2305943042 wait complete. 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.143+0000 7f0a7a92f700 1 Processor -- start 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.143+0000 7f0a7a92f700 1 -- start start 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.143+0000 7f0a7a92f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.143+0000 7f0a7a92f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a74077190 con 0x7f0a74102770 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.143+0000 7f0a73fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.143+0000 7f0a73fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45980/0 (socket says 172.21.15.89:45980) 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.143+0000 7f0a73fff700 1 -- 172.21.15.89:0/1364361922 learned_addr learned my addr 172.21.15.89:0/1364361922 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.143+0000 7f0a73fff700 1 -- 172.21.15.89:0/1364361922 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a680094f0 con 0x7f0a74102770 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.143+0000 7f0a73fff700 1 --2- 172.21.15.89:0/1364361922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74078d10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a68011680 tx=0x7f0a68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.144+0000 7f0a70ff9700 1 -- 172.21.15.89:0/1364361922 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a68010890 con 0x7f0a74102770 2024-02-10T06:08:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.144+0000 7f0a70ff9700 1 -- 172.21.15.89:0/1364361922 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a68010ec0 con 0x7f0a74102770 2024-02-10T06:08:46.959 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.144+0000 7f0a7a92f700 1 -- 172.21.15.89:0/1364361922 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a74077390 con 0x7f0a74102770 2024-02-10T06:08:46.959 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.144+0000 7f0a70ff9700 1 -- 172.21.15.89:0/1364361922 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a680243f0 con 0x7f0a74102770 2024-02-10T06:08:46.959 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.144+0000 7f0a7a92f700 1 -- 172.21.15.89:0/1364361922 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a74077820 con 0x7f0a74102770 2024-02-10T06:08:46.959 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.144+0000 7f0a70ff9700 1 -- 172.21.15.89:0/1364361922 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f0a6802d3f0 con 0x7f0a74102770 2024-02-10T06:08:46.959 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.144+0000 7f0a7a92f700 1 -- 172.21.15.89:0/1364361922 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7404bed0 con 0x7f0a74102770 2024-02-10T06:08:46.959 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.145+0000 7f0a70ff9700 1 --2- 172.21.15.89:0/1364361922 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0a5c036880 0x7f0a5c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:46.959 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.145+0000 7f0a70ff9700 1 -- 172.21.15.89:0/1364361922 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0a6804a690 con 0x7f0a74102770 2024-02-10T06:08:46.959 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.145+0000 7f0a737fe700 1 --2- 172.21.15.89:0/1364361922 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0a5c036880 0x7f0a5c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:46.959 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.145+0000 7f0a737fe700 1 --2- 172.21.15.89:0/1364361922 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0a5c036880 0x7f0a5c038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009990 tx=0x7f0a64006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:46.959 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.149+0000 7f0a70ff9700 1 -- 172.21.15.89:0/1364361922 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0a6802b8e0 con 0x7f0a74102770 2024-02-10T06:08:46.959 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.436+0000 7f0a7a92f700 1 -- 172.21.15.89:0/1364361922 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f0a74077ca0 con 0x7f0a74102770 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.440+0000 7f0a70ff9700 1 -- 172.21.15.89:0/1364361922 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f0a68006d70 con 0x7f0a74102770 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.442+0000 7f0a7a92f700 1 -- 172.21.15.89:0/1364361922 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0a5c036880 msgr2=0x7f0a5c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.442+0000 7f0a7a92f700 1 --2- 172.21.15.89:0/1364361922 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0a5c036880 0x7f0a5c038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009990 tx=0x7f0a64006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.442+0000 7f0a7a92f700 1 -- 172.21.15.89:0/1364361922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 msgr2=0x7f0a74078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.442+0000 7f0a7a92f700 1 --2- 172.21.15.89:0/1364361922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74078d10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a68011680 tx=0x7f0a68009250 comp rx=0 tx=0).stop 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.442+0000 7f0a7a92f700 1 -- 172.21.15.89:0/1364361922 shutdown_connections 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.442+0000 7f0a7a92f700 1 --2- 172.21.15.89:0/1364361922 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0a5c036880 0x7f0a5c038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.442+0000 7f0a7a92f700 1 --2- 172.21.15.89:0/1364361922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a74102770 0x7f0a74078d10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.442+0000 7f0a7a92f700 1 -- 172.21.15.89:0/1364361922 >> 172.21.15.89:0/1364361922 conn(0x7f0a740fded0 msgr2=0x7f0a740ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.443+0000 7f0a7a92f700 1 -- 172.21.15.89:0/1364361922 shutdown_connections 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-02-10T06:08:46.443+0000 7f0a7a92f700 1 -- 172.21.15.89:0/1364361922 wait complete. 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-10T06:08:46.960 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout:Or, if you are only running a single cluster on this host: 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout: ceph telemetry on 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout:For more information see: 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:08:46.961 INFO:teuthology.orchestra.run.smithi089.stdout:Bootstrap complete. 2024-02-10T06:08:46.993 INFO:tasks.cephadm:Fetching config... 2024-02-10T06:08:46.993 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:08:46.993 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T06:08:47.049 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T06:08:47.049 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:08:47.049 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T06:08:47.104 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T06:08:47.104 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:08:47.104 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/keyring of=/dev/stdout 2024-02-10T06:08:47.172 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T06:08:47.172 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:08:47.172 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T06:08:47.226 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T06:08:47.227 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiI1xJj862iCJ/a5TXfqajVgGO70gz8mGFoZtYuyyg51ze7eOXX7jmGDeP+I+O3klk+ORG8k2KLkfB1dWqlK6v4ZbCKGz9UC5A2UtKMO+fXNEd7tULCJxUjriS9Vb+aPNS8oOvFOXYflI6ynR/AaX+vFCPn0NAxcdZh36SRCkH1C7eEtBkPhLF9rx3OUf8IiAY6tLDE+zJpqhR8xQVxfpM2FRLBkdXSSul4DN5PCPlm8NlMvfmImBRSRmP1unXSq+EDjUDyvglawQ8gXM6VwbSC5AGWwYvl8NBfDlIUQEdCOQzsB4XlkRedabNibXtC/HLxGWebglTu8A/LPyxX6pg6iJfCV6U6nZuqoVsPZhTfr7iHyrLzhSW/tWJbaYlcC78G3KWJW4yrsg7pTryHzFe93aSwTvW4q/jBpDqm4nB1QoZF+E+o46qzetrRglHGXXk++ZPXxrD3+Z9z12Gs4HupfrVWmOPO0lwvJSeDmBPYLJxWpKHQdWXSz2mf3t4ztU= ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T06:08:47.304 INFO:teuthology.orchestra.run.smithi089.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiI1xJj862iCJ/a5TXfqajVgGO70gz8mGFoZtYuyyg51ze7eOXX7jmGDeP+I+O3klk+ORG8k2KLkfB1dWqlK6v4ZbCKGz9UC5A2UtKMO+fXNEd7tULCJxUjriS9Vb+aPNS8oOvFOXYflI6ynR/AaX+vFCPn0NAxcdZh36SRCkH1C7eEtBkPhLF9rx3OUf8IiAY6tLDE+zJpqhR8xQVxfpM2FRLBkdXSSul4DN5PCPlm8NlMvfmImBRSRmP1unXSq+EDjUDyvglawQ8gXM6VwbSC5AGWwYvl8NBfDlIUQEdCOQzsB4XlkRedabNibXtC/HLxGWebglTu8A/LPyxX6pg6iJfCV6U6nZuqoVsPZhTfr7iHyrLzhSW/tWJbaYlcC78G3KWJW4yrsg7pTryHzFe93aSwTvW4q/jBpDqm4nB1QoZF+E+o46qzetrRglHGXXk++ZPXxrD3+Z9z12Gs4HupfrVWmOPO0lwvJSeDmBPYLJxWpKHQdWXSz2mf3t4ztU= ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:08:47.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiI1xJj862iCJ/a5TXfqajVgGO70gz8mGFoZtYuyyg51ze7eOXX7jmGDeP+I+O3klk+ORG8k2KLkfB1dWqlK6v4ZbCKGz9UC5A2UtKMO+fXNEd7tULCJxUjriS9Vb+aPNS8oOvFOXYflI6ynR/AaX+vFCPn0NAxcdZh36SRCkH1C7eEtBkPhLF9rx3OUf8IiAY6tLDE+zJpqhR8xQVxfpM2FRLBkdXSSul4DN5PCPlm8NlMvfmImBRSRmP1unXSq+EDjUDyvglawQ8gXM6VwbSC5AGWwYvl8NBfDlIUQEdCOQzsB4XlkRedabNibXtC/HLxGWebglTu8A/LPyxX6pg6iJfCV6U6nZuqoVsPZhTfr7iHyrLzhSW/tWJbaYlcC78G3KWJW4yrsg7pTryHzFe93aSwTvW4q/jBpDqm4nB1QoZF+E+o46qzetrRglHGXXk++ZPXxrD3+Z9z12Gs4HupfrVWmOPO0lwvJSeDmBPYLJxWpKHQdWXSz2mf3t4ztU= ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T06:08:47.365 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiI1xJj862iCJ/a5TXfqajVgGO70gz8mGFoZtYuyyg51ze7eOXX7jmGDeP+I+O3klk+ORG8k2KLkfB1dWqlK6v4ZbCKGz9UC5A2UtKMO+fXNEd7tULCJxUjriS9Vb+aPNS8oOvFOXYflI6ynR/AaX+vFCPn0NAxcdZh36SRCkH1C7eEtBkPhLF9rx3OUf8IiAY6tLDE+zJpqhR8xQVxfpM2FRLBkdXSSul4DN5PCPlm8NlMvfmImBRSRmP1unXSq+EDjUDyvglawQ8gXM6VwbSC5AGWwYvl8NBfDlIUQEdCOQzsB4XlkRedabNibXtC/HLxGWebglTu8A/LPyxX6pg6iJfCV6U6nZuqoVsPZhTfr7iHyrLzhSW/tWJbaYlcC78G3KWJW4yrsg7pTryHzFe93aSwTvW4q/jBpDqm4nB1QoZF+E+o46qzetrRglHGXXk++ZPXxrD3+Z9z12Gs4HupfrVWmOPO0lwvJSeDmBPYLJxWpKHQdWXSz2mf3t4ztU= ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:08:47.380 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T06:08:47.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:47 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1364361922' entity='client.admin' 2024-02-10T06:08:47.631 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:08:50.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:49 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:50.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:49 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:50.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:49 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd/host:smithi089", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:08:50.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:49 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:50.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:49 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:08:50.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:49 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T06:08:50.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:49 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:50.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:49 smithi089 ceph-mon[29217]: Deploying daemon crash.smithi089 on smithi089 2024-02-10T06:08:50.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.465+0000 7f93aea37700 1 -- 172.21.15.89:0/2508087109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a80fae80 msgr2=0x7f93a80fb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:50.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.465+0000 7f93aea37700 1 --2- 172.21.15.89:0/2508087109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a80fae80 0x7f93a80fb290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f93980048c0 tx=0x7f9398009e30 comp rx=0 tx=0).stop 2024-02-10T06:08:50.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.465+0000 7f93aea37700 1 -- 172.21.15.89:0/2508087109 shutdown_connections 2024-02-10T06:08:50.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.465+0000 7f93aea37700 1 --2- 172.21.15.89:0/2508087109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a80fae80 0x7f93a80fb290 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:50.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.465+0000 7f93aea37700 1 -- 172.21.15.89:0/2508087109 >> 172.21.15.89:0/2508087109 conn(0x7f93a80f6920 msgr2=0x7f93a80f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:50.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.465+0000 7f93aea37700 1 -- 172.21.15.89:0/2508087109 shutdown_connections 2024-02-10T06:08:50.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.465+0000 7f93aea37700 1 -- 172.21.15.89:0/2508087109 wait complete. 2024-02-10T06:08:50.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.466+0000 7f93aea37700 1 Processor -- start 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.466+0000 7f93aea37700 1 -- start start 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.466+0000 7f93aea37700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a80fae80 0x7f93a810cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.466+0000 7f93aea37700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a810d400 con 0x7f93a80fae80 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.467+0000 7f93a7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a80fae80 0x7f93a810cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.467+0000 7f93a7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a80fae80 0x7f93a810cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32906/0 (socket says 172.21.15.89:32906) 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.467+0000 7f93a7fff700 1 -- 172.21.15.89:0/3768993981 learned_addr learned my addr 172.21.15.89:0/3768993981 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.467+0000 7f93a7fff700 1 -- 172.21.15.89:0/3768993981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9398005040 con 0x7f93a80fae80 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.467+0000 7f93a7fff700 1 --2- 172.21.15.89:0/3768993981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a80fae80 0x7f93a810cec0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9398009040 tx=0x7f9398009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.467+0000 7f93a4ff9700 1 -- 172.21.15.89:0/3768993981 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f939800bed0 con 0x7f93a80fae80 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.467+0000 7f93a4ff9700 1 -- 172.21.15.89:0/3768993981 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f939800b5c0 con 0x7f93a80fae80 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.467+0000 7f93aea37700 1 -- 172.21.15.89:0/3768993981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a810d600 con 0x7f93a80fae80 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.467+0000 7f93a4ff9700 1 -- 172.21.15.89:0/3768993981 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f939801c5b0 con 0x7f93a80fae80 2024-02-10T06:08:50.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.467+0000 7f93aea37700 1 -- 172.21.15.89:0/3768993981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a810e2a0 con 0x7f93a80fae80 2024-02-10T06:08:50.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.468+0000 7f93a4ff9700 1 -- 172.21.15.89:0/3768993981 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f939801c830 con 0x7f93a80fae80 2024-02-10T06:08:50.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.468+0000 7f93a4ff9700 1 --2- 172.21.15.89:0/3768993981 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f9390036870 0x7f9390038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:50.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.468+0000 7f93aea37700 1 -- 172.21.15.89:0/3768993981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a805f5c0 con 0x7f93a80fae80 2024-02-10T06:08:50.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.468+0000 7f93a4ff9700 1 -- 172.21.15.89:0/3768993981 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9398049ac0 con 0x7f93a80fae80 2024-02-10T06:08:50.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.468+0000 7f93a77fe700 1 --2- 172.21.15.89:0/3768993981 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f9390036870 0x7f9390038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:50.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.469+0000 7f93a77fe700 1 --2- 172.21.15.89:0/3768993981 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f9390036870 0x7f9390038d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f939c009990 tx=0x7f939c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:50.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.473+0000 7f93a4ff9700 1 -- 172.21.15.89:0/3768993981 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9398023300 con 0x7f93a80fae80 2024-02-10T06:08:50.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.662+0000 7f93aea37700 1 -- 172.21.15.89:0/3768993981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f93a804bed0 con 0x7f93a80fae80 2024-02-10T06:08:50.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.665+0000 7f93a4ff9700 1 -- 172.21.15.89:0/3768993981 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9398011b60 con 0x7f93a80fae80 2024-02-10T06:08:50.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.667+0000 7f93aea37700 1 -- 172.21.15.89:0/3768993981 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f9390036870 msgr2=0x7f9390038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:50.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.667+0000 7f93aea37700 1 --2- 172.21.15.89:0/3768993981 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f9390036870 0x7f9390038d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f939c009990 tx=0x7f939c006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:50.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.667+0000 7f93aea37700 1 -- 172.21.15.89:0/3768993981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a80fae80 msgr2=0x7f93a810cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:50.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.667+0000 7f93aea37700 1 --2- 172.21.15.89:0/3768993981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a80fae80 0x7f93a810cec0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9398009040 tx=0x7f9398009e30 comp rx=0 tx=0).stop 2024-02-10T06:08:50.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.667+0000 7f93aea37700 1 -- 172.21.15.89:0/3768993981 shutdown_connections 2024-02-10T06:08:50.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.667+0000 7f93aea37700 1 --2- 172.21.15.89:0/3768993981 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f9390036870 0x7f9390038d20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:50.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.667+0000 7f93aea37700 1 --2- 172.21.15.89:0/3768993981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a80fae80 0x7f93a810cec0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:50.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.667+0000 7f93aea37700 1 -- 172.21.15.89:0/3768993981 >> 172.21.15.89:0/3768993981 conn(0x7f93a80f6920 msgr2=0x7f93a80f7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:50.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.668+0000 7f93aea37700 1 -- 172.21.15.89:0/3768993981 shutdown_connections 2024-02-10T06:08:50.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:50.668+0000 7f93aea37700 1 -- 172.21.15.89:0/3768993981 wait complete. 2024-02-10T06:08:51.460 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-10T06:08:51.460 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-10T06:08:51.772 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:51 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3768993981' entity='client.admin' 2024-02-10T06:08:52.495 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:08:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.935+0000 7f3ef25c8700 1 -- 172.21.15.89:0/2714612757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eec0fcbd0 msgr2=0x7f3eec0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.935+0000 7f3ef25c8700 1 --2- 172.21.15.89:0/2714612757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eec0fcbd0 0x7f3eec0fcfe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0048c0 tx=0x7f3edc009e30 comp rx=0 tx=0).stop 2024-02-10T06:08:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.935+0000 7f3ef25c8700 1 -- 172.21.15.89:0/2714612757 shutdown_connections 2024-02-10T06:08:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.935+0000 7f3ef25c8700 1 --2- 172.21.15.89:0/2714612757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eec0fcbd0 0x7f3eec0fcfe0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:53.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.935+0000 7f3ef25c8700 1 -- 172.21.15.89:0/2714612757 >> 172.21.15.89:0/2714612757 conn(0x7f3eec0f8290 msgr2=0x7f3eec0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.935+0000 7f3ef25c8700 1 -- 172.21.15.89:0/2714612757 shutdown_connections 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.935+0000 7f3ef25c8700 1 -- 172.21.15.89:0/2714612757 wait complete. 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.935+0000 7f3ef25c8700 1 Processor -- start 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.935+0000 7f3ef25c8700 1 -- start start 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.935+0000 7f3ef25c8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eec0fcbd0 0x7f3eec071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.935+0000 7f3ef25c8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec0702e0 con 0x7f3eec0fcbd0 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.936+0000 7f3eebfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eec0fcbd0 0x7f3eec071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.936+0000 7f3eebfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eec0fcbd0 0x7f3eec071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32934/0 (socket says 172.21.15.89:32934) 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.936+0000 7f3eebfff700 1 -- 172.21.15.89:0/1810092210 learned_addr learned my addr 172.21.15.89:0/1810092210 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.936+0000 7f3eebfff700 1 -- 172.21.15.89:0/1810092210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc005040 con 0x7f3eec0fcbd0 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.936+0000 7f3eebfff700 1 --2- 172.21.15.89:0/1810092210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eec0fcbd0 0x7f3eec071e60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3edc009b80 tx=0x7f3edc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.936+0000 7f3ee8ff9700 1 -- 172.21.15.89:0/1810092210 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3edc00b5c0 con 0x7f3eec0fcbd0 2024-02-10T06:08:53.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.936+0000 7f3ef25c8700 1 -- 172.21.15.89:0/1810092210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eec0704e0 con 0x7f3eec0fcbd0 2024-02-10T06:08:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.936+0000 7f3ee8ff9700 1 -- 172.21.15.89:0/1810092210 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3edc00be70 con 0x7f3eec0fcbd0 2024-02-10T06:08:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.936+0000 7f3ee8ff9700 1 -- 172.21.15.89:0/1810092210 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3edc01c660 con 0x7f3eec0fcbd0 2024-02-10T06:08:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.936+0000 7f3ef25c8700 1 -- 172.21.15.89:0/1810092210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eec0709d0 con 0x7f3eec0fcbd0 2024-02-10T06:08:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.937+0000 7f3ee8ff9700 1 -- 172.21.15.89:0/1810092210 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f3edc01c8d0 con 0x7f3eec0fcbd0 2024-02-10T06:08:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.937+0000 7f3ee8ff9700 1 --2- 172.21.15.89:0/1810092210 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3ed4036870 0x7f3ed4038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.937+0000 7f3ef25c8700 1 -- 172.21.15.89:0/1810092210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eec1069a0 con 0x7f3eec0fcbd0 2024-02-10T06:08:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.937+0000 7f3ee8ff9700 1 -- 172.21.15.89:0/1810092210 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3edc049aa0 con 0x7f3eec0fcbd0 2024-02-10T06:08:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.937+0000 7f3eeb7fe700 1 --2- 172.21.15.89:0/1810092210 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3ed4036870 0x7f3ed4038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:53.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.938+0000 7f3eeb7fe700 1 --2- 172.21.15.89:0/1810092210 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3ed4036870 0x7f3ed4038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009990 tx=0x7f3ee0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:53.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:53.942+0000 7f3ee8ff9700 1 -- 172.21.15.89:0/1810092210 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3edc0119c0 con 0x7f3eec0fcbd0 2024-02-10T06:08:54.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.150+0000 7f3ef25c8700 1 -- 172.21.15.89:0/1810092210 --> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f3eec02b3b0 con 0x7f3ed4036870 2024-02-10T06:08:54.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.154+0000 7f3ee8ff9700 1 -- 172.21.15.89:0/1810092210 <== mgr.14162 v2:172.21.15.89:6800/1955086590 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3eec02b3b0 con 0x7f3ed4036870 2024-02-10T06:08:54.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.155+0000 7f3ef25c8700 1 -- 172.21.15.89:0/1810092210 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3ed4036870 msgr2=0x7f3ed4038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.156+0000 7f3ef25c8700 1 --2- 172.21.15.89:0/1810092210 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3ed4036870 0x7f3ed4038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009990 tx=0x7f3ee0006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.156+0000 7f3ef25c8700 1 -- 172.21.15.89:0/1810092210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eec0fcbd0 msgr2=0x7f3eec071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.156+0000 7f3ef25c8700 1 --2- 172.21.15.89:0/1810092210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eec0fcbd0 0x7f3eec071e60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3edc009b80 tx=0x7f3edc009e30 comp rx=0 tx=0).stop 2024-02-10T06:08:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.156+0000 7f3ef25c8700 1 -- 172.21.15.89:0/1810092210 shutdown_connections 2024-02-10T06:08:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.156+0000 7f3ef25c8700 1 --2- 172.21.15.89:0/1810092210 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f3ed4036870 0x7f3ed4038d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.156+0000 7f3ef25c8700 1 --2- 172.21.15.89:0/1810092210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eec0fcbd0 0x7f3eec071e60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.156+0000 7f3ef25c8700 1 -- 172.21.15.89:0/1810092210 >> 172.21.15.89:0/1810092210 conn(0x7f3eec0f8290 msgr2=0x7f3eec0f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.156+0000 7f3ef25c8700 1 -- 172.21.15.89:0/1810092210 shutdown_connections 2024-02-10T06:08:54.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:54.156+0000 7f3ef25c8700 1 -- 172.21.15.89:0/1810092210 wait complete. 2024-02-10T06:08:54.757 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-02-10T06:08:54.758 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:08:54.758 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-02-10T06:08:54.776 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:08:54.776 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:08:54.833 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-02-10T06:08:54.833 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph orch host add smithi173 2024-02-10T06:08:55.100 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:08:55.429 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:55 smithi089 ceph-mon[29217]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:55.429 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:55 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:55.429 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:55 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:55.429 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:55 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:55.430 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:55 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:55.430 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:55 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:56.304 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:56 smithi089 ceph-mon[29217]: Deploying daemon node-exporter.smithi089 on smithi089 2024-02-10T06:08:56.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.506+0000 7f0cca633700 1 -- 172.21.15.89:0/3658635064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cc40fcbc0 msgr2=0x7f0cc40fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:56.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.506+0000 7f0cca633700 1 --2- 172.21.15.89:0/3658635064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cc40fcbc0 0x7f0cc40fcfd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40048c0 tx=0x7f0cb4009e30 comp rx=0 tx=0).stop 2024-02-10T06:08:56.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.506+0000 7f0cca633700 1 -- 172.21.15.89:0/3658635064 shutdown_connections 2024-02-10T06:08:56.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.506+0000 7f0cca633700 1 --2- 172.21.15.89:0/3658635064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cc40fcbc0 0x7f0cc40fcfd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:56.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.506+0000 7f0cca633700 1 -- 172.21.15.89:0/3658635064 >> 172.21.15.89:0/3658635064 conn(0x7f0cc40f82a0 msgr2=0x7f0cc40fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:56.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.507+0000 7f0cca633700 1 -- 172.21.15.89:0/3658635064 shutdown_connections 2024-02-10T06:08:56.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.507+0000 7f0cca633700 1 -- 172.21.15.89:0/3658635064 wait complete. 2024-02-10T06:08:56.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.507+0000 7f0cca633700 1 Processor -- start 2024-02-10T06:08:56.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.507+0000 7f0cca633700 1 -- start start 2024-02-10T06:08:56.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.508+0000 7f0cca633700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cc40fcbc0 0x7f0cc4071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:56.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.508+0000 7f0cca633700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc40702e0 con 0x7f0cc40fcbc0 2024-02-10T06:08:56.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.508+0000 7f0cc3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cc40fcbc0 0x7f0cc4071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:56.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.508+0000 7f0cc3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cc40fcbc0 0x7f0cc4071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32962/0 (socket says 172.21.15.89:32962) 2024-02-10T06:08:56.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.508+0000 7f0cc3fff700 1 -- 172.21.15.89:0/2658758395 learned_addr learned my addr 172.21.15.89:0/2658758395 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:08:56.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.508+0000 7f0cc3fff700 1 -- 172.21.15.89:0/2658758395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb4005040 con 0x7f0cc40fcbc0 2024-02-10T06:08:56.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.508+0000 7f0cc3fff700 1 --2- 172.21.15.89:0/2658758395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cc40fcbc0 0x7f0cc4071e60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4009680 tx=0x7f0cb4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:56.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.508+0000 7f0cc0ff9700 1 -- 172.21.15.89:0/2658758395 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cb400bed0 con 0x7f0cc40fcbc0 2024-02-10T06:08:56.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.509+0000 7f0cc0ff9700 1 -- 172.21.15.89:0/2658758395 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cb400b5c0 con 0x7f0cc40fcbc0 2024-02-10T06:08:56.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.509+0000 7f0cc0ff9700 1 -- 172.21.15.89:0/2658758395 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cb401c5b0 con 0x7f0cc40fcbc0 2024-02-10T06:08:56.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.509+0000 7f0cca633700 1 -- 172.21.15.89:0/2658758395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc40704e0 con 0x7f0cc40fcbc0 2024-02-10T06:08:56.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.509+0000 7f0cca633700 1 -- 172.21.15.89:0/2658758395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc4070910 con 0x7f0cc40fcbc0 2024-02-10T06:08:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.509+0000 7f0cc0ff9700 1 -- 172.21.15.89:0/2658758395 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f0cb401c7d0 con 0x7f0cc40fcbc0 2024-02-10T06:08:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.510+0000 7f0cc0ff9700 1 --2- 172.21.15.89:0/2658758395 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0cac036870 0x7f0cac038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:08:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.510+0000 7f0cc0ff9700 1 -- 172.21.15.89:0/2658758395 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0cb404aa80 con 0x7f0cc40fcbc0 2024-02-10T06:08:56.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.510+0000 7f0cca633700 1 -- 172.21.15.89:0/2658758395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc4106990 con 0x7f0cc40fcbc0 2024-02-10T06:08:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.510+0000 7f0cc37fe700 1 --2- 172.21.15.89:0/2658758395 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0cac036870 0x7f0cac038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:08:56.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.510+0000 7f0cc37fe700 1 --2- 172.21.15.89:0/2658758395 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0cac036870 0x7f0cac038d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009990 tx=0x7f0cb8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:08:56.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.515+0000 7f0cc0ff9700 1 -- 172.21.15.89:0/2658758395 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0cb4011db0 con 0x7f0cc40fcbc0 2024-02-10T06:08:56.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:56.718+0000 7f0cca633700 1 -- 172.21.15.89:0/2658758395 --> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cc402b340 con 0x7f0cac036870 2024-02-10T06:08:57.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:57 smithi089 ceph-mon[29217]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:08:58.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:58.755+0000 7f0cc0ff9700 1 -- 172.21.15.89:0/2658758395 <== mgr.14162 v2:172.21.15.89:6800/1955086590 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f0cc402b340 con 0x7f0cac036870 2024-02-10T06:08:58.756 INFO:teuthology.orchestra.run.smithi089.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-02-10T06:08:58.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:58.757+0000 7f0cca633700 1 -- 172.21.15.89:0/2658758395 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0cac036870 msgr2=0x7f0cac038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:58.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:58.757+0000 7f0cca633700 1 --2- 172.21.15.89:0/2658758395 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0cac036870 0x7f0cac038d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009990 tx=0x7f0cb8006e40 comp rx=0 tx=0).stop 2024-02-10T06:08:58.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:58.757+0000 7f0cca633700 1 -- 172.21.15.89:0/2658758395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cc40fcbc0 msgr2=0x7f0cc4071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:08:58.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:58.757+0000 7f0cca633700 1 --2- 172.21.15.89:0/2658758395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cc40fcbc0 0x7f0cc4071e60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4009680 tx=0x7f0cb4009e30 comp rx=0 tx=0).stop 2024-02-10T06:08:58.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:58.757+0000 7f0cca633700 1 -- 172.21.15.89:0/2658758395 shutdown_connections 2024-02-10T06:08:58.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:58.757+0000 7f0cca633700 1 --2- 172.21.15.89:0/2658758395 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f0cac036870 0x7f0cac038d20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:58.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:58.757+0000 7f0cca633700 1 --2- 172.21.15.89:0/2658758395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cc40fcbc0 0x7f0cc4071e60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:08:58.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:58.757+0000 7f0cca633700 1 -- 172.21.15.89:0/2658758395 >> 172.21.15.89:0/2658758395 conn(0x7f0cc40f82a0 msgr2=0x7f0cc40f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:08:58.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:58.758+0000 7f0cca633700 1 -- 172.21.15.89:0/2658758395 shutdown_connections 2024-02-10T06:08:58.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:08:58.758+0000 7f0cca633700 1 -- 172.21.15.89:0/2658758395 wait complete. 2024-02-10T06:08:59.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:58 smithi089 ceph-mon[29217]: Deploying cephadm binary to smithi173 2024-02-10T06:08:59.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:58 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:59.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:58 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:08:59.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T06:09:00.049 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:09:00.174 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:59 smithi089 ceph-mon[29217]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:00.174 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:08:59 smithi089 ceph-mon[29217]: Added host smithi173 2024-02-10T06:09:01.203 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:01 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:01.204 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:01 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:01.204 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:01 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:01.204 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:01 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:01.204 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:01 smithi089 ceph-mon[29217]: Deploying daemon alertmanager.smithi089 on smithi089 2024-02-10T06:09:01.204 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:01 smithi089 ceph-mon[29217]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.418+0000 7f36a81ea700 1 -- 172.21.15.89:0/2417294587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a00faea0 msgr2=0x7f36a00fb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.418+0000 7f36a81ea700 1 --2- 172.21.15.89:0/2417294587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a00faea0 0x7f36a00fb2b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f369c0048c0 tx=0x7f369c009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.418+0000 7f36a81ea700 1 -- 172.21.15.89:0/2417294587 shutdown_connections 2024-02-10T06:09:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.418+0000 7f36a81ea700 1 --2- 172.21.15.89:0/2417294587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a00faea0 0x7f36a00fb2b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.418+0000 7f36a81ea700 1 -- 172.21.15.89:0/2417294587 >> 172.21.15.89:0/2417294587 conn(0x7f36a00f6920 msgr2=0x7f36a00f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.419+0000 7f36a81ea700 1 -- 172.21.15.89:0/2417294587 shutdown_connections 2024-02-10T06:09:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.419+0000 7f36a81ea700 1 -- 172.21.15.89:0/2417294587 wait complete. 2024-02-10T06:09:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.419+0000 7f36a81ea700 1 Processor -- start 2024-02-10T06:09:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.419+0000 7f36a81ea700 1 -- start start 2024-02-10T06:09:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.419+0000 7f36a81ea700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a00faea0 0x7f36a010cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.419+0000 7f36a81ea700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a010d450 con 0x7f36a00faea0 2024-02-10T06:09:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.419+0000 7f36a5f86700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a00faea0 0x7f36a010cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.419+0000 7f36a5f86700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a00faea0 0x7f36a010cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36904/0 (socket says 172.21.15.89:36904) 2024-02-10T06:09:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.419+0000 7f36a5f86700 1 -- 172.21.15.89:0/1491194526 learned_addr learned my addr 172.21.15.89:0/1491194526 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:09:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.420+0000 7f36a5f86700 1 -- 172.21.15.89:0/1491194526 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c005040 con 0x7f36a00faea0 2024-02-10T06:09:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.420+0000 7f36a5f86700 1 --2- 172.21.15.89:0/1491194526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a00faea0 0x7f36a010cf10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f369c009330 tx=0x7f369c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.420+0000 7f36967fc700 1 -- 172.21.15.89:0/1491194526 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f369c004e60 con 0x7f36a00faea0 2024-02-10T06:09:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.420+0000 7f36967fc700 1 -- 172.21.15.89:0/1491194526 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f369c0096b0 con 0x7f36a00faea0 2024-02-10T06:09:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.420+0000 7f36a81ea700 1 -- 172.21.15.89:0/1491194526 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a010d650 con 0x7f36a00faea0 2024-02-10T06:09:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.420+0000 7f36967fc700 1 -- 172.21.15.89:0/1491194526 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f369c01c5b0 con 0x7f36a00faea0 2024-02-10T06:09:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.420+0000 7f36a81ea700 1 -- 172.21.15.89:0/1491194526 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a010e2f0 con 0x7f36a00faea0 2024-02-10T06:09:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.421+0000 7f36967fc700 1 -- 172.21.15.89:0/1491194526 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f369c013980 con 0x7f36a00faea0 2024-02-10T06:09:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.421+0000 7f36a81ea700 1 -- 172.21.15.89:0/1491194526 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a005f5c0 con 0x7f36a00faea0 2024-02-10T06:09:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.421+0000 7f36967fc700 1 --2- 172.21.15.89:0/1491194526 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f368c0368c0 0x7f368c038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.421+0000 7f36967fc700 1 -- 172.21.15.89:0/1491194526 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f369c049bc0 con 0x7f36a00faea0 2024-02-10T06:09:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.421+0000 7f36a5785700 1 --2- 172.21.15.89:0/1491194526 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f368c0368c0 0x7f368c038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.422+0000 7f36a5785700 1 --2- 172.21.15.89:0/1491194526 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f368c0368c0 0x7f368c038d70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3690009990 tx=0x7f3690006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.426+0000 7f36967fc700 1 -- 172.21.15.89:0/1491194526 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f369c011550 con 0x7f36a00faea0 2024-02-10T06:09:01.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.656+0000 7f36a81ea700 1 -- 172.21.15.89:0/1491194526 --> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36a010eeb0 con 0x7f368c0368c0 2024-02-10T06:09:01.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.657+0000 7f36967fc700 1 -- 172.21.15.89:0/1491194526 <== mgr.14162 v2:172.21.15.89:6800/1955086590 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f36a010eeb0 con 0x7f368c0368c0 2024-02-10T06:09:01.657 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:09:01.658 INFO:teuthology.orchestra.run.smithi089.stdout:[{"addr": "172.21.15.89", "hostname": "smithi089", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-02-10T06:09:01.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.659+0000 7f36a81ea700 1 -- 172.21.15.89:0/1491194526 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f368c0368c0 msgr2=0x7f368c038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:01.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.659+0000 7f36a81ea700 1 --2- 172.21.15.89:0/1491194526 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f368c0368c0 0x7f368c038d70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3690009990 tx=0x7f3690006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:01.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.659+0000 7f36a81ea700 1 -- 172.21.15.89:0/1491194526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a00faea0 msgr2=0x7f36a010cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:01.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.659+0000 7f36a81ea700 1 --2- 172.21.15.89:0/1491194526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a00faea0 0x7f36a010cf10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f369c009330 tx=0x7f369c009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:01.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.659+0000 7f36a81ea700 1 -- 172.21.15.89:0/1491194526 shutdown_connections 2024-02-10T06:09:01.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.659+0000 7f36a81ea700 1 --2- 172.21.15.89:0/1491194526 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f368c0368c0 0x7f368c038d70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:01.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.659+0000 7f36a81ea700 1 --2- 172.21.15.89:0/1491194526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a00faea0 0x7f36a010cf10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:01.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.659+0000 7f36a81ea700 1 -- 172.21.15.89:0/1491194526 >> 172.21.15.89:0/1491194526 conn(0x7f36a00f6920 msgr2=0x7f36a00f76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:01.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.659+0000 7f36a81ea700 1 -- 172.21.15.89:0/1491194526 shutdown_connections 2024-02-10T06:09:01.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:01.659+0000 7f36a81ea700 1 -- 172.21.15.89:0/1491194526 wait complete. 2024-02-10T06:09:02.240 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:02 smithi089 ceph-mon[29217]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:09:02.264 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T06:09:02.264 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T06:09:02.510 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:09:04.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:03 smithi089 ceph-mon[29217]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:04.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:03 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:05.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.630+0000 7efcb7d89700 1 -- 172.21.15.89:0/2956235315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efcb00fbac0 msgr2=0x7efcb00fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.630+0000 7efcb7d89700 1 --2- 172.21.15.89:0/2956235315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efcb00fbac0 0x7efcb00fbed0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efcac0048c0 tx=0x7efcac009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.630+0000 7efcb7d89700 1 -- 172.21.15.89:0/2956235315 shutdown_connections 2024-02-10T06:09:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.630+0000 7efcb7d89700 1 --2- 172.21.15.89:0/2956235315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efcb00fbac0 0x7efcb00fbed0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.630+0000 7efcb7d89700 1 -- 172.21.15.89:0/2956235315 >> 172.21.15.89:0/2956235315 conn(0x7efcb00f71a0 msgr2=0x7efcb00f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.630+0000 7efcb7d89700 1 -- 172.21.15.89:0/2956235315 shutdown_connections 2024-02-10T06:09:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.631+0000 7efcb7d89700 1 -- 172.21.15.89:0/2956235315 wait complete. 2024-02-10T06:09:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.631+0000 7efcb7d89700 1 Processor -- start 2024-02-10T06:09:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.631+0000 7efcb7d89700 1 -- start start 2024-02-10T06:09:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.631+0000 7efcb7d89700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efcb00fbac0 0x7efcb00f9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.631+0000 7efcb7d89700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00f7690 con 0x7efcb00fbac0 2024-02-10T06:09:05.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.631+0000 7efcb5b25700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efcb00fbac0 0x7efcb00f9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.631+0000 7efcb5b25700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efcb00fbac0 0x7efcb00f9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36928/0 (socket says 172.21.15.89:36928) 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.631+0000 7efcb5b25700 1 -- 172.21.15.89:0/828227090 learned_addr learned my addr 172.21.15.89:0/828227090 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.632+0000 7efcb5b25700 1 -- 172.21.15.89:0/828227090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac005040 con 0x7efcb00fbac0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.632+0000 7efcb5b25700 1 --2- 172.21.15.89:0/828227090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efcb00fbac0 0x7efcb00f9210 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efcac009b80 tx=0x7efcac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.632+0000 7efca67fc700 1 -- 172.21.15.89:0/828227090 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcac00b5c0 con 0x7efcb00fbac0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.632+0000 7efcb7d89700 1 -- 172.21.15.89:0/828227090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb00f7890 con 0x7efcb00fbac0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.632+0000 7efca67fc700 1 -- 172.21.15.89:0/828227090 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efcac00be70 con 0x7efcb00fbac0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.632+0000 7efca67fc700 1 -- 172.21.15.89:0/828227090 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcac01c660 con 0x7efcb00fbac0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.632+0000 7efcb7d89700 1 -- 172.21.15.89:0/828227090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb00f7d80 con 0x7efcb00fbac0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.633+0000 7efca67fc700 1 -- 172.21.15.89:0/828227090 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7efcac013420 con 0x7efcb00fbac0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.633+0000 7efcb7d89700 1 -- 172.21.15.89:0/828227090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb01068a0 con 0x7efcb00fbac0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.633+0000 7efca67fc700 1 --2- 172.21.15.89:0/828227090 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7efc9c036870 0x7efc9c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.633+0000 7efca67fc700 1 -- 172.21.15.89:0/828227090 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7efcac0499c0 con 0x7efcb00fbac0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.633+0000 7efcb5324700 1 --2- 172.21.15.89:0/828227090 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7efc9c036870 0x7efc9c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:05.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.633+0000 7efcb5324700 1 --2- 172.21.15.89:0/828227090 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7efc9c036870 0x7efc9c038d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efca0009990 tx=0x7efca0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:05.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.638+0000 7efca67fc700 1 -- 172.21.15.89:0/828227090 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7efcac011550 con 0x7efcb00fbac0 2024-02-10T06:09:05.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.829+0000 7efcb7d89700 1 -- 172.21.15.89:0/828227090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7efcb004bed0 con 0x7efcb00fbac0 2024-02-10T06:09:05.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.926+0000 7efca67fc700 1 -- 172.21.15.89:0/828227090 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7efcac011370 con 0x7efcb00fbac0 2024-02-10T06:09:05.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.928+0000 7efcb7d89700 1 -- 172.21.15.89:0/828227090 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7efc9c036870 msgr2=0x7efc9c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:05.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.928+0000 7efcb7d89700 1 --2- 172.21.15.89:0/828227090 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7efc9c036870 0x7efc9c038d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efca0009990 tx=0x7efca0006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:05.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.928+0000 7efcb7d89700 1 -- 172.21.15.89:0/828227090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efcb00fbac0 msgr2=0x7efcb00f9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:05.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.928+0000 7efcb7d89700 1 --2- 172.21.15.89:0/828227090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efcb00fbac0 0x7efcb00f9210 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efcac009b80 tx=0x7efcac009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:05.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.928+0000 7efcb7d89700 1 -- 172.21.15.89:0/828227090 shutdown_connections 2024-02-10T06:09:05.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.928+0000 7efcb7d89700 1 --2- 172.21.15.89:0/828227090 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7efc9c036870 0x7efc9c038d20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:05.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.928+0000 7efcb7d89700 1 --2- 172.21.15.89:0/828227090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efcb00fbac0 0x7efcb00f9210 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:05.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.928+0000 7efcb7d89700 1 -- 172.21.15.89:0/828227090 >> 172.21.15.89:0/828227090 conn(0x7efcb00f71a0 msgr2=0x7efcb0066410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:05.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.928+0000 7efcb7d89700 1 -- 172.21.15.89:0/828227090 shutdown_connections 2024-02-10T06:09:05.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:09:05.928+0000 7efcb7d89700 1 -- 172.21.15.89:0/828227090 wait complete. 2024-02-10T06:09:05.930 INFO:teuthology.orchestra.run.smithi089.stderr:adjusted tunables profile to default 2024-02-10T06:09:06.244 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:05 smithi089 ceph-mon[29217]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:06.244 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:05 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/828227090' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T06:09:07.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:06 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/828227090' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T06:09:07.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:06 smithi089 ceph-mon[29217]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T06:09:08.071 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:07 smithi089 ceph-mon[29217]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:08.535 INFO:tasks.cephadm:Adding mon.smithi089 on smithi089 2024-02-10T06:09:08.536 INFO:tasks.cephadm:Adding mon.smithi173 on smithi173 2024-02-10T06:09:08.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph orch apply mon '2;smithi089:172.21.15.89=smithi089;smithi173:172.21.15.173=smithi173' 2024-02-10T06:09:08.786 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:08.832 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:10.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:09 smithi089 ceph-mon[29217]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:10.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.375+0000 7fa749cf2700 1 -- 172.21.15.173:0/3754598341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7440fbad0 msgr2=0x7fa7440fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:10.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.375+0000 7fa749cf2700 1 --2- 172.21.15.173:0/3754598341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7440fbad0 0x7fa7440fbee0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa7340048c0 tx=0x7fa734009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:10.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.375+0000 7fa749cf2700 1 -- 172.21.15.173:0/3754598341 shutdown_connections 2024-02-10T06:09:10.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.375+0000 7fa749cf2700 1 --2- 172.21.15.173:0/3754598341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7440fbad0 0x7fa7440fbee0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:10.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.375+0000 7fa749cf2700 1 -- 172.21.15.173:0/3754598341 >> 172.21.15.173:0/3754598341 conn(0x7fa7440f7190 msgr2=0x7fa7440f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:10.378 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.375+0000 7fa749cf2700 1 -- 172.21.15.173:0/3754598341 shutdown_connections 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.375+0000 7fa749cf2700 1 -- 172.21.15.173:0/3754598341 wait complete. 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.375+0000 7fa749cf2700 1 Processor -- start 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.375+0000 7fa749cf2700 1 -- start start 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.375+0000 7fa749cf2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7440fbad0 0x7fa74410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.375+0000 7fa749cf2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74410d400 con 0x7fa7440fbad0 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.376+0000 7fa7437fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7440fbad0 0x7fa74410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.376+0000 7fa7437fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7440fbad0 0x7fa74410cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:51162/0 (socket says 172.21.15.173:51162) 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.376+0000 7fa7437fe700 1 -- 172.21.15.173:0/2403892465 learned_addr learned my addr 172.21.15.173:0/2403892465 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.376+0000 7fa7437fe700 1 -- 172.21.15.173:0/2403892465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa734005040 con 0x7fa7440fbad0 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.376+0000 7fa7437fe700 1 --2- 172.21.15.173:0/2403892465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7440fbad0 0x7fa74410cec0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa734009680 tx=0x7fa734009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.376+0000 7fa740ff9700 1 -- 172.21.15.173:0/2403892465 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa73400bed0 con 0x7fa7440fbad0 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.376+0000 7fa749cf2700 1 -- 172.21.15.173:0/2403892465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74410d600 con 0x7fa7440fbad0 2024-02-10T06:09:10.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.376+0000 7fa740ff9700 1 -- 172.21.15.173:0/2403892465 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa73400b5c0 con 0x7fa7440fbad0 2024-02-10T06:09:10.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.376+0000 7fa740ff9700 1 -- 172.21.15.173:0/2403892465 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa73401c5b0 con 0x7fa7440fbad0 2024-02-10T06:09:10.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.376+0000 7fa749cf2700 1 -- 172.21.15.173:0/2403892465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74410e2a0 con 0x7fa7440fbad0 2024-02-10T06:09:10.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.377+0000 7fa740ff9700 1 -- 172.21.15.173:0/2403892465 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fa73401c7d0 con 0x7fa7440fbad0 2024-02-10T06:09:10.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.377+0000 7fa740ff9700 1 --2- 172.21.15.173:0/2403892465 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fa730036870 0x7fa730038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:10.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.377+0000 7fa740ff9700 1 -- 172.21.15.173:0/2403892465 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa734049b00 con 0x7fa7440fbad0 2024-02-10T06:09:10.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.377+0000 7fa749cf2700 1 -- 172.21.15.173:0/2403892465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74405f5c0 con 0x7fa7440fbad0 2024-02-10T06:09:10.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.378+0000 7fa73bfff700 1 --2- 172.21.15.173:0/2403892465 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fa730036870 0x7fa730038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:10.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.378+0000 7fa73bfff700 1 --2- 172.21.15.173:0/2403892465 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fa730036870 0x7fa730038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009990 tx=0x7fa72c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:10.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.382+0000 7fa740ff9700 1 -- 172.21.15.173:0/2403892465 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa734023300 con 0x7fa7440fbad0 2024-02-10T06:09:10.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.571+0000 7fa749cf2700 1 -- 172.21.15.173:0/2403892465 --> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi089:172.21.15.89=smithi089;smithi173:172.21.15.173=smithi173", "target": ["mon-mgr", ""]}) v1 -- 0x7fa74410e650 con 0x7fa730036870 2024-02-10T06:09:10.577 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.576+0000 7fa740ff9700 1 -- 172.21.15.173:0/2403892465 <== mgr.14162 v2:172.21.15.89:6800/1955086590 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa74410e650 con 0x7fa730036870 2024-02-10T06:09:10.577 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-02-10T06:09:10.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.578+0000 7fa749cf2700 1 -- 172.21.15.173:0/2403892465 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fa730036870 msgr2=0x7fa730038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:10.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.578+0000 7fa749cf2700 1 --2- 172.21.15.173:0/2403892465 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fa730036870 0x7fa730038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009990 tx=0x7fa72c006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:10.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.578+0000 7fa749cf2700 1 -- 172.21.15.173:0/2403892465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7440fbad0 msgr2=0x7fa74410cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:10.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.578+0000 7fa749cf2700 1 --2- 172.21.15.173:0/2403892465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7440fbad0 0x7fa74410cec0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa734009680 tx=0x7fa734009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:10.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.579+0000 7fa749cf2700 1 -- 172.21.15.173:0/2403892465 shutdown_connections 2024-02-10T06:09:10.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.579+0000 7fa749cf2700 1 --2- 172.21.15.173:0/2403892465 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fa730036870 0x7fa730038d20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:10.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.579+0000 7fa749cf2700 1 --2- 172.21.15.173:0/2403892465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7440fbad0 0x7fa74410cec0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:10.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.579+0000 7fa749cf2700 1 -- 172.21.15.173:0/2403892465 >> 172.21.15.173:0/2403892465 conn(0x7fa7440f7190 msgr2=0x7fa744065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:10.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.579+0000 7fa749cf2700 1 -- 172.21.15.173:0/2403892465 shutdown_connections 2024-02-10T06:09:10.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:10.579+0000 7fa749cf2700 1 -- 172.21.15.173:0/2403892465 wait complete. 2024-02-10T06:09:11.145 DEBUG:teuthology.orchestra.run.smithi173:mon.smithi173> sudo journalctl -f -n 0 -u ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mon.smithi173.service 2024-02-10T06:09:11.148 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:11.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:11.422 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:11.469 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:11.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:11.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:11.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:11.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:11.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:11.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:11.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:09:11.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:09:11.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:11.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: Deploying daemon grafana.smithi089 on smithi089 2024-02-10T06:09:11.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:11.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi089:172.21.15.89=smithi089;smithi173:172.21.15.173=smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:09:11.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: Saving service mon spec with placement smithi089:172.21.15.89=smithi089;smithi173:172.21.15.173=smithi173;count:2 2024-02-10T06:09:11.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:11 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:12.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.901+0000 7f62aa057700 1 -- 172.21.15.173:0/531766304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40fcbc0 msgr2=0x7f62a40fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:12.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.901+0000 7f62aa057700 1 --2- 172.21.15.173:0/531766304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40fcbc0 0x7f62a40fcfd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f62940048c0 tx=0x7f6294009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:12.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.901+0000 7f62aa057700 1 -- 172.21.15.173:0/531766304 shutdown_connections 2024-02-10T06:09:12.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.901+0000 7f62aa057700 1 --2- 172.21.15.173:0/531766304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40fcbc0 0x7f62a40fcfd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:12.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.901+0000 7f62aa057700 1 -- 172.21.15.173:0/531766304 >> 172.21.15.173:0/531766304 conn(0x7f62a40f82a0 msgr2=0x7f62a40fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:12.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.901+0000 7f62aa057700 1 -- 172.21.15.173:0/531766304 shutdown_connections 2024-02-10T06:09:12.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.901+0000 7f62aa057700 1 -- 172.21.15.173:0/531766304 wait complete. 2024-02-10T06:09:12.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.902+0000 7f62aa057700 1 Processor -- start 2024-02-10T06:09:12.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.902+0000 7f62aa057700 1 -- start start 2024-02-10T06:09:12.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.902+0000 7f62aa057700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40fcbc0 0x7f62a410cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:12.903 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.902+0000 7f62aa057700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a410d420 con 0x7f62a40fcbc0 2024-02-10T06:09:12.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.902+0000 7f62a37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40fcbc0 0x7f62a410cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.902+0000 7f62a37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40fcbc0 0x7f62a410cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:51186/0 (socket says 172.21.15.173:51186) 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.902+0000 7f62a37fe700 1 -- 172.21.15.173:0/3529416868 learned_addr learned my addr 172.21.15.173:0/3529416868 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.903+0000 7f62a37fe700 1 -- 172.21.15.173:0/3529416868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6294005040 con 0x7f62a40fcbc0 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.903+0000 7f62a37fe700 1 --2- 172.21.15.173:0/3529416868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40fcbc0 0x7f62a410cee0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6294009680 tx=0x7f6294009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.903+0000 7f628bfff700 1 -- 172.21.15.173:0/3529416868 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f629400bed0 con 0x7f62a40fcbc0 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.903+0000 7f628bfff700 1 -- 172.21.15.173:0/3529416868 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f629400b5c0 con 0x7f62a40fcbc0 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.903+0000 7f62aa057700 1 -- 172.21.15.173:0/3529416868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a410d620 con 0x7f62a40fcbc0 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.903+0000 7f628bfff700 1 -- 172.21.15.173:0/3529416868 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f629401c5b0 con 0x7f62a40fcbc0 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.903+0000 7f62aa057700 1 -- 172.21.15.173:0/3529416868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a410e2c0 con 0x7f62a40fcbc0 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.904+0000 7f628bfff700 1 -- 172.21.15.173:0/3529416868 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f629401c7d0 con 0x7f62a40fcbc0 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.904+0000 7f62aa057700 1 -- 172.21.15.173:0/3529416868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a405f5c0 con 0x7f62a40fcbc0 2024-02-10T06:09:12.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.904+0000 7f628bfff700 1 --2- 172.21.15.173:0/3529416868 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f628c036870 0x7f628c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:12.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.904+0000 7f628bfff700 1 -- 172.21.15.173:0/3529416868 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6294049a90 con 0x7f62a40fcbc0 2024-02-10T06:09:12.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.904+0000 7f62a2ffd700 1 --2- 172.21.15.173:0/3529416868 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f628c036870 0x7f628c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:12.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.905+0000 7f62a2ffd700 1 --2- 172.21.15.173:0/3529416868 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f628c036870 0x7f628c038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6298009990 tx=0x7f6298006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:12.909 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:12.909+0000 7f628bfff700 1 -- 172.21.15.173:0/3529416868 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6294025400 con 0x7f62a40fcbc0 2024-02-10T06:09:13.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.165+0000 7f62aa057700 1 -- 172.21.15.173:0/3529416868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62a410ee10 con 0x7f62a40fcbc0 2024-02-10T06:09:13.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.166+0000 7f628bfff700 1 -- 172.21.15.173:0/3529416868 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6294011370 con 0x7f62a40fcbc0 2024-02-10T06:09:13.167 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:13.167 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:13.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.168+0000 7f62aa057700 1 -- 172.21.15.173:0/3529416868 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f628c036870 msgr2=0x7f628c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:13.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.168+0000 7f62aa057700 1 --2- 172.21.15.173:0/3529416868 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f628c036870 0x7f628c038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6298009990 tx=0x7f6298006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:13.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.168+0000 7f62aa057700 1 -- 172.21.15.173:0/3529416868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40fcbc0 msgr2=0x7f62a410cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:13.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.168+0000 7f62aa057700 1 --2- 172.21.15.173:0/3529416868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40fcbc0 0x7f62a410cee0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6294009680 tx=0x7f6294009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:13.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.168+0000 7f62aa057700 1 -- 172.21.15.173:0/3529416868 shutdown_connections 2024-02-10T06:09:13.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.168+0000 7f62aa057700 1 --2- 172.21.15.173:0/3529416868 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f628c036870 0x7f628c038d20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:13.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.168+0000 7f62aa057700 1 --2- 172.21.15.173:0/3529416868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a40fcbc0 0x7f62a410cee0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:13.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.168+0000 7f62aa057700 1 -- 172.21.15.173:0/3529416868 >> 172.21.15.173:0/3529416868 conn(0x7f62a40f82a0 msgr2=0x7f62a40f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:13.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.168+0000 7f62aa057700 1 -- 172.21.15.173:0/3529416868 shutdown_connections 2024-02-10T06:09:13.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:13.168+0000 7f62aa057700 1 -- 172.21.15.173:0/3529416868 wait complete. 2024-02-10T06:09:13.170 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:14.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:13 smithi089 ceph-mon[29217]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:14.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:13 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/3529416868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:14.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:13 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:14.773 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:14.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:15.022 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:15.069 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:16.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:15 smithi089 ceph-mon[29217]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:16.639 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.637+0000 7f47245d9700 1 -- 172.21.15.173:0/132701436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f471c0fbac0 msgr2=0x7f471c0fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:16.639 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.637+0000 7f47245d9700 1 --2- 172.21.15.173:0/132701436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f471c0fbac0 0x7f471c0fbed0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f47180048c0 tx=0x7f4718009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:16.639 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.637+0000 7f47245d9700 1 -- 172.21.15.173:0/132701436 shutdown_connections 2024-02-10T06:09:16.639 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.637+0000 7f47245d9700 1 --2- 172.21.15.173:0/132701436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f471c0fbac0 0x7f471c0fbed0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:16.639 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.637+0000 7f47245d9700 1 -- 172.21.15.173:0/132701436 >> 172.21.15.173:0/132701436 conn(0x7f471c0f71a0 msgr2=0x7f471c0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:16.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.637+0000 7f47245d9700 1 -- 172.21.15.173:0/132701436 shutdown_connections 2024-02-10T06:09:16.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.637+0000 7f47245d9700 1 -- 172.21.15.173:0/132701436 wait complete. 2024-02-10T06:09:16.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.637+0000 7f47245d9700 1 Processor -- start 2024-02-10T06:09:16.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.637+0000 7f47245d9700 1 -- start start 2024-02-10T06:09:16.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.637+0000 7f47245d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f471c0fbac0 0x7f471c10fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:16.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.637+0000 7f47245d9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c066830 con 0x7f471c0fbac0 2024-02-10T06:09:16.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.638+0000 7f4722375700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f471c0fbac0 0x7f471c10fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:16.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.638+0000 7f4722375700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f471c0fbac0 0x7f471c10fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:50018/0 (socket says 172.21.15.173:50018) 2024-02-10T06:09:16.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.638+0000 7f4722375700 1 -- 172.21.15.173:0/3754811506 learned_addr learned my addr 172.21.15.173:0/3754811506 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:16.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.638+0000 7f4722375700 1 -- 172.21.15.173:0/3754811506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4718005040 con 0x7f471c0fbac0 2024-02-10T06:09:16.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.638+0000 7f4722375700 1 --2- 172.21.15.173:0/3754811506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f471c0fbac0 0x7f471c10fbe0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4718009680 tx=0x7f4718009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:16.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.638+0000 7f4712ffd700 1 -- 172.21.15.173:0/3754811506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f471800bed0 con 0x7f471c0fbac0 2024-02-10T06:09:16.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.638+0000 7f47245d9700 1 -- 172.21.15.173:0/3754811506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c066a30 con 0x7f471c0fbac0 2024-02-10T06:09:16.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.638+0000 7f4712ffd700 1 -- 172.21.15.173:0/3754811506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f471800b5c0 con 0x7f471c0fbac0 2024-02-10T06:09:16.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.638+0000 7f4712ffd700 1 -- 172.21.15.173:0/3754811506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f471801c5b0 con 0x7f471c0fbac0 2024-02-10T06:09:16.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.638+0000 7f47245d9700 1 -- 172.21.15.173:0/3754811506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c110360 con 0x7f471c0fbac0 2024-02-10T06:09:16.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.639+0000 7f4712ffd700 1 -- 172.21.15.173:0/3754811506 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f471801c820 con 0x7f471c0fbac0 2024-02-10T06:09:16.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.639+0000 7f4712ffd700 1 --2- 172.21.15.173:0/3754811506 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f4708036820 0x7f4708038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:16.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.639+0000 7f47245d9700 1 -- 172.21.15.173:0/3754811506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c105890 con 0x7f471c0fbac0 2024-02-10T06:09:16.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.639+0000 7f4712ffd700 1 -- 172.21.15.173:0/3754811506 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4718049ce0 con 0x7f471c0fbac0 2024-02-10T06:09:16.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.640+0000 7f4721b74700 1 --2- 172.21.15.173:0/3754811506 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f4708036820 0x7f4708038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:16.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.641+0000 7f4721b74700 1 --2- 172.21.15.173:0/3754811506 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f4708036820 0x7f4708038cd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f470c009990 tx=0x7f470c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:16.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.645+0000 7f4712ffd700 1 -- 172.21.15.173:0/3754811506 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4718011dd0 con 0x7f471c0fbac0 2024-02-10T06:09:16.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.898+0000 7f47245d9700 1 -- 172.21.15.173:0/3754811506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f471c101880 con 0x7f471c0fbac0 2024-02-10T06:09:16.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.899+0000 7f4712ffd700 1 -- 172.21.15.173:0/3754811506 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4718011370 con 0x7f471c0fbac0 2024-02-10T06:09:16.899 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:16.899 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:16.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.900+0000 7f47245d9700 1 -- 172.21.15.173:0/3754811506 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f4708036820 msgr2=0x7f4708038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:16.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.900+0000 7f47245d9700 1 --2- 172.21.15.173:0/3754811506 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f4708036820 0x7f4708038cd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f470c009990 tx=0x7f470c006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:16.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.900+0000 7f47245d9700 1 -- 172.21.15.173:0/3754811506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f471c0fbac0 msgr2=0x7f471c10fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:16.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.900+0000 7f47245d9700 1 --2- 172.21.15.173:0/3754811506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f471c0fbac0 0x7f471c10fbe0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4718009680 tx=0x7f4718009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:16.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.901+0000 7f47245d9700 1 -- 172.21.15.173:0/3754811506 shutdown_connections 2024-02-10T06:09:16.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.901+0000 7f47245d9700 1 --2- 172.21.15.173:0/3754811506 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f4708036820 0x7f4708038cd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:16.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.901+0000 7f47245d9700 1 --2- 172.21.15.173:0/3754811506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f471c0fbac0 0x7f471c10fbe0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:16.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.901+0000 7f47245d9700 1 -- 172.21.15.173:0/3754811506 >> 172.21.15.173:0/3754811506 conn(0x7f471c0f71a0 msgr2=0x7f471c0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:16.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.901+0000 7f47245d9700 1 -- 172.21.15.173:0/3754811506 shutdown_connections 2024-02-10T06:09:16.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:16.901+0000 7f47245d9700 1 -- 172.21.15.173:0/3754811506 wait complete. 2024-02-10T06:09:16.902 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:17.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:16 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/3754811506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:18.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:17 smithi089 ceph-mon[29217]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:18.477 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:18.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:18.732 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:18.781 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:20.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.250+0000 7f5334239700 1 -- 172.21.15.173:0/2341226889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c066880 msgr2=0x7f532c0ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:20.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.250+0000 7f5334239700 1 --2- 172.21.15.173:0/2341226889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c066880 0x7f532c0ff790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f53280048c0 tx=0x7f5328009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:20.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.250+0000 7f5334239700 1 -- 172.21.15.173:0/2341226889 shutdown_connections 2024-02-10T06:09:20.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.250+0000 7f5334239700 1 --2- 172.21.15.173:0/2341226889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c066880 0x7f532c0ff790 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:20.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.250+0000 7f5334239700 1 -- 172.21.15.173:0/2341226889 >> 172.21.15.173:0/2341226889 conn(0x7f532c0f71a0 msgr2=0x7f532c0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:20.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.250+0000 7f5334239700 1 -- 172.21.15.173:0/2341226889 shutdown_connections 2024-02-10T06:09:20.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.250+0000 7f5334239700 1 -- 172.21.15.173:0/2341226889 wait complete. 2024-02-10T06:09:20.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.250+0000 7f5334239700 1 Processor -- start 2024-02-10T06:09:20.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f5334239700 1 -- start start 2024-02-10T06:09:20.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f5334239700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c066880 0x7f532c10ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:20.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f5334239700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c10d3c0 con 0x7f532c066880 2024-02-10T06:09:20.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f5331fd5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c066880 0x7f532c10ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:20.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f5331fd5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c066880 0x7f532c10ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:50030/0 (socket says 172.21.15.173:50030) 2024-02-10T06:09:20.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f5331fd5700 1 -- 172.21.15.173:0/168033500 learned_addr learned my addr 172.21.15.173:0/168033500 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:20.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f5331fd5700 1 -- 172.21.15.173:0/168033500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5328005040 con 0x7f532c066880 2024-02-10T06:09:20.253 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f5331fd5700 1 --2- 172.21.15.173:0/168033500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c066880 0x7f532c10ce80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5328009680 tx=0x7f5328009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f53227fc700 1 -- 172.21.15.173:0/168033500 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f532800b5c0 con 0x7f532c066880 2024-02-10T06:09:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f5334239700 1 -- 172.21.15.173:0/168033500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532c10d5c0 con 0x7f532c066880 2024-02-10T06:09:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f53227fc700 1 -- 172.21.15.173:0/168033500 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5328004e60 con 0x7f532c066880 2024-02-10T06:09:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.251+0000 7f53227fc700 1 -- 172.21.15.173:0/168033500 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f532801c590 con 0x7f532c066880 2024-02-10T06:09:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.252+0000 7f5334239700 1 -- 172.21.15.173:0/168033500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532c10e260 con 0x7f532c066880 2024-02-10T06:09:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.252+0000 7f53227fc700 1 -- 172.21.15.173:0/168033500 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f532801c810 con 0x7f532c066880 2024-02-10T06:09:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.252+0000 7f5334239700 1 -- 172.21.15.173:0/168033500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532c05f5c0 con 0x7f532c066880 2024-02-10T06:09:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.252+0000 7f53227fc700 1 --2- 172.21.15.173:0/168033500 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f5318036870 0x7f5318038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.253+0000 7f53227fc700 1 -- 172.21.15.173:0/168033500 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5328049d40 con 0x7f532c066880 2024-02-10T06:09:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.253+0000 7f53317d4700 1 --2- 172.21.15.173:0/168033500 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f5318036870 0x7f5318038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.253+0000 7f53317d4700 1 --2- 172.21.15.173:0/168033500 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f5318036870 0x7f5318038d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f531c009990 tx=0x7f531c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:20.257 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.257+0000 7f53227fc700 1 -- 172.21.15.173:0/168033500 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5328050080 con 0x7f532c066880 2024-02-10T06:09:20.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:19 smithi089 ceph-mon[29217]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:20.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.510+0000 7f5334239700 1 -- 172.21.15.173:0/168033500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f532c10edb0 con 0x7f532c066880 2024-02-10T06:09:20.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.510+0000 7f53227fc700 1 -- 172.21.15.173:0/168033500 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5328011550 con 0x7f532c066880 2024-02-10T06:09:20.511 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:20.511 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:20.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.512+0000 7f5334239700 1 -- 172.21.15.173:0/168033500 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f5318036870 msgr2=0x7f5318038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:20.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.512+0000 7f5334239700 1 --2- 172.21.15.173:0/168033500 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f5318036870 0x7f5318038d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f531c009990 tx=0x7f531c006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:20.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.512+0000 7f5334239700 1 -- 172.21.15.173:0/168033500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c066880 msgr2=0x7f532c10ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:20.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.512+0000 7f5334239700 1 --2- 172.21.15.173:0/168033500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c066880 0x7f532c10ce80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5328009680 tx=0x7f5328009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:20.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.512+0000 7f5334239700 1 -- 172.21.15.173:0/168033500 shutdown_connections 2024-02-10T06:09:20.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.512+0000 7f5334239700 1 --2- 172.21.15.173:0/168033500 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f5318036870 0x7f5318038d20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:20.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.512+0000 7f5334239700 1 --2- 172.21.15.173:0/168033500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f532c066880 0x7f532c10ce80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:20.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.512+0000 7f5334239700 1 -- 172.21.15.173:0/168033500 >> 172.21.15.173:0/168033500 conn(0x7f532c0f71a0 msgr2=0x7f532c0f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:20.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.512+0000 7f5334239700 1 -- 172.21.15.173:0/168033500 shutdown_connections 2024-02-10T06:09:20.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:20.512+0000 7f5334239700 1 -- 172.21.15.173:0/168033500 wait complete. 2024-02-10T06:09:20.514 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:21.275 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:20 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/168033500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:22.007 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:22.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:22.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:21 smithi089 ceph-mon[29217]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:22.257 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:22.304 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:23.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.797+0000 7fc73281d700 1 -- 172.21.15.173:0/4208840137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc72c0fafc0 msgr2=0x7fc72c0fb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:23.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.797+0000 7fc73281d700 1 --2- 172.21.15.173:0/4208840137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc72c0fafc0 0x7fc72c0fb3d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc71c0048c0 tx=0x7fc71c009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.797+0000 7fc73281d700 1 -- 172.21.15.173:0/4208840137 shutdown_connections 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.797+0000 7fc73281d700 1 --2- 172.21.15.173:0/4208840137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc72c0fafc0 0x7fc72c0fb3d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.797+0000 7fc73281d700 1 -- 172.21.15.173:0/4208840137 >> 172.21.15.173:0/4208840137 conn(0x7fc72c0f69c0 msgr2=0x7fc72c0f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.797+0000 7fc73281d700 1 -- 172.21.15.173:0/4208840137 shutdown_connections 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.797+0000 7fc73281d700 1 -- 172.21.15.173:0/4208840137 wait complete. 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.797+0000 7fc73281d700 1 Processor -- start 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.798+0000 7fc73281d700 1 -- start start 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.798+0000 7fc73281d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc72c0fafc0 0x7fc72c10d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.798+0000 7fc73281d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c10d6a0 con 0x7fc72c0fafc0 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.798+0000 7fc72bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc72c0fafc0 0x7fc72c10d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.798+0000 7fc72bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc72c0fafc0 0x7fc72c10d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:50058/0 (socket says 172.21.15.173:50058) 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.798+0000 7fc72bfff700 1 -- 172.21.15.173:0/1017078856 learned_addr learned my addr 172.21.15.173:0/1017078856 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:23.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.798+0000 7fc72bfff700 1 -- 172.21.15.173:0/1017078856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c005040 con 0x7fc72c0fafc0 2024-02-10T06:09:23.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.798+0000 7fc72bfff700 1 --2- 172.21.15.173:0/1017078856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc72c0fafc0 0x7fc72c10d160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009300 tx=0x7fc71c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:23.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.799+0000 7fc728ff9700 1 -- 172.21.15.173:0/1017078856 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc71c004e60 con 0x7fc72c0fafc0 2024-02-10T06:09:23.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.799+0000 7fc73281d700 1 -- 172.21.15.173:0/1017078856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72c10d8a0 con 0x7fc72c0fafc0 2024-02-10T06:09:23.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.799+0000 7fc728ff9700 1 -- 172.21.15.173:0/1017078856 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc71c009680 con 0x7fc72c0fafc0 2024-02-10T06:09:23.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.799+0000 7fc728ff9700 1 -- 172.21.15.173:0/1017078856 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc71c01c5b0 con 0x7fc72c0fafc0 2024-02-10T06:09:23.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.799+0000 7fc73281d700 1 -- 172.21.15.173:0/1017078856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72c10e540 con 0x7fc72c0fafc0 2024-02-10T06:09:23.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.799+0000 7fc728ff9700 1 -- 172.21.15.173:0/1017078856 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fc71c013420 con 0x7fc72c0fafc0 2024-02-10T06:09:23.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.799+0000 7fc73281d700 1 -- 172.21.15.173:0/1017078856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72c05f5c0 con 0x7fc72c0fafc0 2024-02-10T06:09:23.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.800+0000 7fc728ff9700 1 --2- 172.21.15.173:0/1017078856 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc714036870 0x7fc714038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:23.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.800+0000 7fc728ff9700 1 -- 172.21.15.173:0/1017078856 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc71c049b20 con 0x7fc72c0fafc0 2024-02-10T06:09:23.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.800+0000 7fc72b7fe700 1 --2- 172.21.15.173:0/1017078856 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc714036870 0x7fc714038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:23.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.800+0000 7fc72b7fe700 1 --2- 172.21.15.173:0/1017078856 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc714036870 0x7fc714038d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc720009990 tx=0x7fc720006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:23.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:23.805+0000 7fc728ff9700 1 -- 172.21.15.173:0/1017078856 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc71c013980 con 0x7fc72c0fafc0 2024-02-10T06:09:23.988 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:23 smithi089 ceph-mon[29217]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:23.988 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:23 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:23.988 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:23 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:23.988 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:23 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:23.989 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:23 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:23.989 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:23 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:23.989 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:23 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:24.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.062+0000 7fc73281d700 1 -- 172.21.15.173:0/1017078856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc72c10ef50 con 0x7fc72c0fafc0 2024-02-10T06:09:24.063 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.063+0000 7fc728ff9700 1 -- 172.21.15.173:0/1017078856 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc71c011370 con 0x7fc72c0fafc0 2024-02-10T06:09:24.063 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:24.063 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:24.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.065+0000 7fc73281d700 1 -- 172.21.15.173:0/1017078856 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc714036870 msgr2=0x7fc714038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:24.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.065+0000 7fc73281d700 1 --2- 172.21.15.173:0/1017078856 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc714036870 0x7fc714038d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc720009990 tx=0x7fc720006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:24.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.065+0000 7fc73281d700 1 -- 172.21.15.173:0/1017078856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc72c0fafc0 msgr2=0x7fc72c10d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:24.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.065+0000 7fc73281d700 1 --2- 172.21.15.173:0/1017078856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc72c0fafc0 0x7fc72c10d160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009300 tx=0x7fc71c009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:24.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.065+0000 7fc73281d700 1 -- 172.21.15.173:0/1017078856 shutdown_connections 2024-02-10T06:09:24.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.065+0000 7fc73281d700 1 --2- 172.21.15.173:0/1017078856 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc714036870 0x7fc714038d20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:24.065 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.065+0000 7fc73281d700 1 --2- 172.21.15.173:0/1017078856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc72c0fafc0 0x7fc72c10d160 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:24.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.065+0000 7fc73281d700 1 -- 172.21.15.173:0/1017078856 >> 172.21.15.173:0/1017078856 conn(0x7fc72c0f69c0 msgr2=0x7fc72c0f77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:24.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.065+0000 7fc73281d700 1 -- 172.21.15.173:0/1017078856 shutdown_connections 2024-02-10T06:09:24.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:24.065+0000 7fc73281d700 1 -- 172.21.15.173:0/1017078856 wait complete. 2024-02-10T06:09:24.066 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:25.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:24 smithi089 ceph-mon[29217]: Deploying daemon prometheus.smithi089 on smithi089 2024-02-10T06:09:25.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:24 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/1017078856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:25.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:25.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:25.929 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:25.976 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:26.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:25 smithi089 ceph-mon[29217]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:27.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.401+0000 7fda268ea700 1 -- 172.21.15.173:0/2654062822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda200fcbb0 msgr2=0x7fda200fcfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:27.469 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.401+0000 7fda268ea700 1 --2- 172.21.15.173:0/2654062822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda200fcbb0 0x7fda200fcfc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fda100048c0 tx=0x7fda10009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:27.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.401+0000 7fda268ea700 1 -- 172.21.15.173:0/2654062822 shutdown_connections 2024-02-10T06:09:27.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.401+0000 7fda268ea700 1 --2- 172.21.15.173:0/2654062822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda200fcbb0 0x7fda200fcfc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:27.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.401+0000 7fda268ea700 1 -- 172.21.15.173:0/2654062822 >> 172.21.15.173:0/2654062822 conn(0x7fda200f8330 msgr2=0x7fda200fa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:27.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.401+0000 7fda268ea700 1 -- 172.21.15.173:0/2654062822 shutdown_connections 2024-02-10T06:09:27.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.401+0000 7fda268ea700 1 -- 172.21.15.173:0/2654062822 wait complete. 2024-02-10T06:09:27.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.402+0000 7fda268ea700 1 Processor -- start 2024-02-10T06:09:27.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.402+0000 7fda268ea700 1 -- start start 2024-02-10T06:09:27.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.402+0000 7fda268ea700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda200fcbb0 0x7fda20071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:27.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.402+0000 7fda268ea700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200702e0 con 0x7fda200fcbb0 2024-02-10T06:09:27.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.402+0000 7fda1ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda200fcbb0 0x7fda20071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.402+0000 7fda1ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda200fcbb0 0x7fda20071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:48540/0 (socket says 172.21.15.173:48540) 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.402+0000 7fda1ffff700 1 -- 172.21.15.173:0/3074145562 learned_addr learned my addr 172.21.15.173:0/3074145562 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.403+0000 7fda1ffff700 1 -- 172.21.15.173:0/3074145562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda10005040 con 0x7fda200fcbb0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.403+0000 7fda1ffff700 1 --2- 172.21.15.173:0/3074145562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda200fcbb0 0x7fda20071e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fda10009680 tx=0x7fda10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.403+0000 7fda1cff9700 1 -- 172.21.15.173:0/3074145562 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda1000b5c0 con 0x7fda200fcbb0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.403+0000 7fda1cff9700 1 -- 172.21.15.173:0/3074145562 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda10004e60 con 0x7fda200fcbb0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.403+0000 7fda268ea700 1 -- 172.21.15.173:0/3074145562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda200704e0 con 0x7fda200fcbb0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.403+0000 7fda1cff9700 1 -- 172.21.15.173:0/3074145562 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda1001c590 con 0x7fda200fcbb0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.403+0000 7fda268ea700 1 -- 172.21.15.173:0/3074145562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda200709d0 con 0x7fda200fcbb0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.404+0000 7fda1cff9700 1 -- 172.21.15.173:0/3074145562 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fda10013420 con 0x7fda200fcbb0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.404+0000 7fda268ea700 1 -- 172.21.15.173:0/3074145562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda20106900 con 0x7fda200fcbb0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.404+0000 7fda1cff9700 1 --2- 172.21.15.173:0/3074145562 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fda08036870 0x7fda08038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.404+0000 7fda1cff9700 1 -- 172.21.15.173:0/3074145562 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fda10049b70 con 0x7fda200fcbb0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.404+0000 7fda1f7fe700 1 --2- 172.21.15.173:0/3074145562 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fda08036870 0x7fda08038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:27.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.405+0000 7fda1f7fe700 1 --2- 172.21.15.173:0/3074145562 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fda08036870 0x7fda08038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fda14009990 tx=0x7fda14006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:27.474 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.409+0000 7fda1cff9700 1 -- 172.21.15.173:0/3074145562 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fda10025400 con 0x7fda200fcbb0 2024-02-10T06:09:27.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.666+0000 7fda268ea700 1 -- 172.21.15.173:0/3074145562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda2004bed0 con 0x7fda200fcbb0 2024-02-10T06:09:27.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.667+0000 7fda1cff9700 1 -- 172.21.15.173:0/3074145562 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fda10011370 con 0x7fda200fcbb0 2024-02-10T06:09:27.667 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:27.667 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:27.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.668+0000 7fda268ea700 1 -- 172.21.15.173:0/3074145562 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fda08036870 msgr2=0x7fda08038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:27.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.668+0000 7fda268ea700 1 --2- 172.21.15.173:0/3074145562 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fda08036870 0x7fda08038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fda14009990 tx=0x7fda14006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:27.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.668+0000 7fda268ea700 1 -- 172.21.15.173:0/3074145562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda200fcbb0 msgr2=0x7fda20071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:27.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.668+0000 7fda268ea700 1 --2- 172.21.15.173:0/3074145562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda200fcbb0 0x7fda20071e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fda10009680 tx=0x7fda10009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:27.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.669+0000 7fda268ea700 1 -- 172.21.15.173:0/3074145562 shutdown_connections 2024-02-10T06:09:27.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.669+0000 7fda268ea700 1 --2- 172.21.15.173:0/3074145562 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fda08036870 0x7fda08038d20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:27.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.669+0000 7fda268ea700 1 --2- 172.21.15.173:0/3074145562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda200fcbb0 0x7fda20071e60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:27.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.669+0000 7fda268ea700 1 -- 172.21.15.173:0/3074145562 >> 172.21.15.173:0/3074145562 conn(0x7fda200f8330 msgr2=0x7fda200f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:27.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.669+0000 7fda268ea700 1 -- 172.21.15.173:0/3074145562 shutdown_connections 2024-02-10T06:09:27.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:27.669+0000 7fda268ea700 1 -- 172.21.15.173:0/3074145562 wait complete. 2024-02-10T06:09:27.670 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:28.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:27 smithi089 ceph-mon[29217]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:28.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:27 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/3074145562' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:29.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:29.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:29.480 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:29.526 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:30.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:29 smithi089 ceph-mon[29217]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:30.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:29 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:31.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.015+0000 7fc9f7c91700 1 -- 172.21.15.173:0/2259257611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9f00fc0d0 msgr2=0x7fc9f00fc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:31.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.015+0000 7fc9f7c91700 1 --2- 172.21.15.173:0/2259257611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9f00fc0d0 0x7fc9f00fc4e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0048c0 tx=0x7fc9ec009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:31.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.015+0000 7fc9f7c91700 1 -- 172.21.15.173:0/2259257611 shutdown_connections 2024-02-10T06:09:31.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.015+0000 7fc9f7c91700 1 --2- 172.21.15.173:0/2259257611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9f00fc0d0 0x7fc9f00fc4e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:31.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.015+0000 7fc9f7c91700 1 -- 172.21.15.173:0/2259257611 >> 172.21.15.173:0/2259257611 conn(0x7fc9f00f69c0 msgr2=0x7fc9f00f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:31.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.015+0000 7fc9f7c91700 1 -- 172.21.15.173:0/2259257611 shutdown_connections 2024-02-10T06:09:31.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.015+0000 7fc9f7c91700 1 -- 172.21.15.173:0/2259257611 wait complete. 2024-02-10T06:09:31.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.016+0000 7fc9f7c91700 1 Processor -- start 2024-02-10T06:09:31.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.016+0000 7fc9f7c91700 1 -- start start 2024-02-10T06:09:31.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.016+0000 7fc9f7c91700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9f00fc0d0 0x7fc9f00f8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:31.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.016+0000 7fc9f7c91700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f00f6f30 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.016+0000 7fc9f5a2d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9f00fc0d0 0x7fc9f00f8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:31.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.016+0000 7fc9f5a2d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9f00fc0d0 0x7fc9f00f8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:48552/0 (socket says 172.21.15.173:48552) 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.016+0000 7fc9f5a2d700 1 -- 172.21.15.173:0/1175970889 learned_addr learned my addr 172.21.15.173:0/1175970889 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.017+0000 7fc9f5a2d700 1 -- 172.21.15.173:0/1175970889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ec005040 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.017+0000 7fc9f5a2d700 1 --2- 172.21.15.173:0/1175970889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9f00fc0d0 0x7fc9f00f8ab0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec009300 tx=0x7fc9ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.017+0000 7fc9e67fc700 1 -- 172.21.15.173:0/1175970889 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9ec004e60 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.017+0000 7fc9f7c91700 1 -- 172.21.15.173:0/1175970889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f00f7130 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.017+0000 7fc9e67fc700 1 -- 172.21.15.173:0/1175970889 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9ec009680 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.017+0000 7fc9e67fc700 1 -- 172.21.15.173:0/1175970889 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9ec01c5b0 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.017+0000 7fc9f7c91700 1 -- 172.21.15.173:0/1175970889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f00f7620 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.018+0000 7fc9e67fc700 1 -- 172.21.15.173:0/1175970889 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fc9ec01c870 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.018+0000 7fc9e67fc700 1 --2- 172.21.15.173:0/1175970889 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc9dc036870 0x7fc9dc038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.018+0000 7fc9f7c91700 1 -- 172.21.15.173:0/1175970889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f0106810 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.018+0000 7fc9e67fc700 1 -- 172.21.15.173:0/1175970889 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc9ec049b40 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.020 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.018+0000 7fc9f522c700 1 --2- 172.21.15.173:0/1175970889 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc9dc036870 0x7fc9dc038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:31.020 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.019+0000 7fc9f522c700 1 --2- 172.21.15.173:0/1175970889 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc9dc036870 0x7fc9dc038d20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0009990 tx=0x7fc9e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:31.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.023+0000 7fc9e67fc700 1 -- 172.21.15.173:0/1175970889 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc9ec023300 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.281 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.280+0000 7fc9f7c91700 1 -- 172.21.15.173:0/1175970889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9f005f5c0 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.281 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.281+0000 7fc9e67fc700 1 -- 172.21.15.173:0/1175970889 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc9ec03a030 con 0x7fc9f00fc0d0 2024-02-10T06:09:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:31.281 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:31.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.283+0000 7fc9f7c91700 1 -- 172.21.15.173:0/1175970889 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc9dc036870 msgr2=0x7fc9dc038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:31.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.283+0000 7fc9f7c91700 1 --2- 172.21.15.173:0/1175970889 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc9dc036870 0x7fc9dc038d20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0009990 tx=0x7fc9e0006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:31.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.283+0000 7fc9f7c91700 1 -- 172.21.15.173:0/1175970889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9f00fc0d0 msgr2=0x7fc9f00f8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:31.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.283+0000 7fc9f7c91700 1 --2- 172.21.15.173:0/1175970889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9f00fc0d0 0x7fc9f00f8ab0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec009300 tx=0x7fc9ec009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:31.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.283+0000 7fc9f7c91700 1 -- 172.21.15.173:0/1175970889 shutdown_connections 2024-02-10T06:09:31.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.283+0000 7fc9f7c91700 1 --2- 172.21.15.173:0/1175970889 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc9dc036870 0x7fc9dc038d20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:31.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.283+0000 7fc9f7c91700 1 --2- 172.21.15.173:0/1175970889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc9f00fc0d0 0x7fc9f00f8ab0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:31.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.283+0000 7fc9f7c91700 1 -- 172.21.15.173:0/1175970889 >> 172.21.15.173:0/1175970889 conn(0x7fc9f00f69c0 msgr2=0x7fc9f00fb600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:31.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.283+0000 7fc9f7c91700 1 -- 172.21.15.173:0/1175970889 shutdown_connections 2024-02-10T06:09:31.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:31.283+0000 7fc9f7c91700 1 -- 172.21.15.173:0/1175970889 wait complete. 2024-02-10T06:09:31.284 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:31.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:31 smithi089 ceph-mon[29217]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:31.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:31 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/1175970889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:32.787 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:32.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:33.038 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:33.086 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:34.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:33 smithi089 ceph-mon[29217]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:34.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.595+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/3834503423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf68066880 msgr2=0x7fbf680ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:34.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.595+0000 7fbf6fb8a700 1 --2- 172.21.15.173:0/3834503423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf68066880 0x7fbf680ff790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbf580048c0 tx=0x7fbf58009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:34.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.595+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/3834503423 shutdown_connections 2024-02-10T06:09:34.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.595+0000 7fbf6fb8a700 1 --2- 172.21.15.173:0/3834503423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf68066880 0x7fbf680ff790 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:34.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.595+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/3834503423 >> 172.21.15.173:0/3834503423 conn(0x7fbf680f71a0 msgr2=0x7fbf680f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:34.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.596+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/3834503423 shutdown_connections 2024-02-10T06:09:34.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.596+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/3834503423 wait complete. 2024-02-10T06:09:34.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.596+0000 7fbf6fb8a700 1 Processor -- start 2024-02-10T06:09:34.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.596+0000 7fbf6fb8a700 1 -- start start 2024-02-10T06:09:34.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.596+0000 7fbf6fb8a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf68066880 0x7fbf6810f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:34.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.596+0000 7fbf6fb8a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6810f5d0 con 0x7fbf68066880 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.596+0000 7fbf6d926700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf68066880 0x7fbf6810f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.597+0000 7fbf6d926700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf68066880 0x7fbf6810f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:48564/0 (socket says 172.21.15.173:48564) 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.597+0000 7fbf6d926700 1 -- 172.21.15.173:0/176553744 learned_addr learned my addr 172.21.15.173:0/176553744 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.597+0000 7fbf6d926700 1 -- 172.21.15.173:0/176553744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf58005040 con 0x7fbf68066880 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.597+0000 7fbf6d926700 1 --2- 172.21.15.173:0/176553744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf68066880 0x7fbf6810f090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009680 tx=0x7fbf58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.597+0000 7fbf5e7fc700 1 -- 172.21.15.173:0/176553744 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf5800b5c0 con 0x7fbf68066880 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.597+0000 7fbf5e7fc700 1 -- 172.21.15.173:0/176553744 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf58004e60 con 0x7fbf68066880 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.597+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/176553744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6810f7d0 con 0x7fbf68066880 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.597+0000 7fbf5e7fc700 1 -- 172.21.15.173:0/176553744 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf5801c590 con 0x7fbf68066880 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.597+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/176553744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf68110470 con 0x7fbf68066880 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.598+0000 7fbf5e7fc700 1 -- 172.21.15.173:0/176553744 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fbf5801c810 con 0x7fbf68066880 2024-02-10T06:09:34.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.598+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/176553744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6805f5c0 con 0x7fbf68066880 2024-02-10T06:09:34.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.598+0000 7fbf5e7fc700 1 --2- 172.21.15.173:0/176553744 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fbf54036870 0x7fbf54038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:34.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.598+0000 7fbf5e7fc700 1 -- 172.21.15.173:0/176553744 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbf58049bb0 con 0x7fbf68066880 2024-02-10T06:09:34.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.598+0000 7fbf6d125700 1 --2- 172.21.15.173:0/176553744 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fbf54036870 0x7fbf54038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:34.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.599+0000 7fbf6d125700 1 --2- 172.21.15.173:0/176553744 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fbf54036870 0x7fbf54038d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009990 tx=0x7fbf64006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:34.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.603+0000 7fbf5e7fc700 1 -- 172.21.15.173:0/176553744 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbf58011550 con 0x7fbf68066880 2024-02-10T06:09:34.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.855+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/176553744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf68110fc0 con 0x7fbf68066880 2024-02-10T06:09:34.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.856+0000 7fbf5e7fc700 1 -- 172.21.15.173:0/176553744 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbf58011370 con 0x7fbf68066880 2024-02-10T06:09:34.857 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:34.858 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:34.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.858+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/176553744 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fbf54036870 msgr2=0x7fbf54038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:34.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.858+0000 7fbf6fb8a700 1 --2- 172.21.15.173:0/176553744 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fbf54036870 0x7fbf54038d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009990 tx=0x7fbf64006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:34.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.858+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/176553744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf68066880 msgr2=0x7fbf6810f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:34.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.858+0000 7fbf6fb8a700 1 --2- 172.21.15.173:0/176553744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf68066880 0x7fbf6810f090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009680 tx=0x7fbf58009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:34.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.858+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/176553744 shutdown_connections 2024-02-10T06:09:34.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.858+0000 7fbf6fb8a700 1 --2- 172.21.15.173:0/176553744 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fbf54036870 0x7fbf54038d20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:34.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.858+0000 7fbf6fb8a700 1 --2- 172.21.15.173:0/176553744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf68066880 0x7fbf6810f090 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:34.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.858+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/176553744 >> 172.21.15.173:0/176553744 conn(0x7fbf680f71a0 msgr2=0x7fbf680f95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:34.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.858+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/176553744 shutdown_connections 2024-02-10T06:09:34.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:34.858+0000 7fbf6fb8a700 1 -- 172.21.15.173:0/176553744 wait complete. 2024-02-10T06:09:34.860 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:35.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:35 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:35.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:35 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:35.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:35 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:35.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:35 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T06:09:35.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:35 smithi089 ceph-mon[29217]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:09:35.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:35 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/176553744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:36.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:36.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:36.651 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:36.698 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:36.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:36 smithi089 ceph-mon[29217]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T06:09:36.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:36 smithi089 ceph-mon[29217]: mgrmap e14: smithi089.pwmrye(active, since 57s) 2024-02-10T06:09:38.144 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.141+0000 7f29c0466700 1 -- 172.21.15.173:0/350328806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29b80fd130 msgr2=0x7f29b80fd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.141+0000 7f29c0466700 1 --2- 172.21.15.173:0/350328806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29b80fd130 0x7f29b80fd540 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f29b40048c0 tx=0x7f29b4009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.142+0000 7f29c0466700 1 -- 172.21.15.173:0/350328806 shutdown_connections 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.142+0000 7f29c0466700 1 --2- 172.21.15.173:0/350328806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29b80fd130 0x7f29b80fd540 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.142+0000 7f29c0466700 1 -- 172.21.15.173:0/350328806 >> 172.21.15.173:0/350328806 conn(0x7f29b80f69c0 msgr2=0x7f29b80f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.142+0000 7f29c0466700 1 -- 172.21.15.173:0/350328806 shutdown_connections 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.142+0000 7f29c0466700 1 -- 172.21.15.173:0/350328806 wait complete. 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.142+0000 7f29c0466700 1 Processor -- start 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.142+0000 7f29c0466700 1 -- start start 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.142+0000 7f29c0466700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29b80fd130 0x7f29b810f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.142+0000 7f29c0466700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b810f660 con 0x7f29b80fd130 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.143+0000 7f29be202700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29b80fd130 0x7f29b810f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:38.145 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.143+0000 7f29be202700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29b80fd130 0x7f29b810f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:55166/0 (socket says 172.21.15.173:55166) 2024-02-10T06:09:38.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.143+0000 7f29be202700 1 -- 172.21.15.173:0/91469994 learned_addr learned my addr 172.21.15.173:0/91469994 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:38.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.143+0000 7f29be202700 1 -- 172.21.15.173:0/91469994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b4005040 con 0x7f29b80fd130 2024-02-10T06:09:38.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.143+0000 7f29be202700 1 --2- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29b80fd130 0x7f29b810f120 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009330 tx=0x7f29b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:38.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.143+0000 7f29aeffd700 1 -- 172.21.15.173:0/91469994 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b400be50 con 0x7f29b80fd130 2024-02-10T06:09:38.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.143+0000 7f29aeffd700 1 -- 172.21.15.173:0/91469994 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29b40096b0 con 0x7f29b80fd130 2024-02-10T06:09:38.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.143+0000 7f29c0466700 1 -- 172.21.15.173:0/91469994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b810f860 con 0x7f29b80fd130 2024-02-10T06:09:38.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.143+0000 7f29aeffd700 1 -- 172.21.15.173:0/91469994 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b401c5b0 con 0x7f29b80fd130 2024-02-10T06:09:38.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.143+0000 7f29c0466700 1 -- 172.21.15.173:0/91469994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b8110500 con 0x7f29b80fd130 2024-02-10T06:09:38.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.144+0000 7f29aeffd700 1 -- 172.21.15.173:0/91469994 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7f29b401c870 con 0x7f29b80fd130 2024-02-10T06:09:38.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.144+0000 7f29c0466700 1 -- 172.21.15.173:0/91469994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b805f5c0 con 0x7f29b80fd130 2024-02-10T06:09:38.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.144+0000 7f29aeffd700 1 --2- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f29a40368c0 0x7f29a4038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:38.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.144+0000 7f29aeffd700 1 -- 172.21.15.173:0/91469994 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f29b4049ae0 con 0x7f29b80fd130 2024-02-10T06:09:38.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.144+0000 7f29bda01700 1 -- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f29a40368c0 msgr2=0x7f29a4038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1955086590 2024-02-10T06:09:38.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.144+0000 7f29bda01700 1 --2- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f29a40368c0 0x7f29a4038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:09:38.150 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.149+0000 7f29aeffd700 1 -- 172.21.15.173:0/91469994 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f29b4023300 con 0x7f29b80fd130 2024-02-10T06:09:38.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.345+0000 7f29bda01700 1 -- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f29a40368c0 msgr2=0x7f29a4038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1955086590 2024-02-10T06:09:38.347 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.345+0000 7f29bda01700 1 --2- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f29a40368c0 0x7f29a4038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:09:38.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.405+0000 7f29c0466700 1 -- 172.21.15.173:0/91469994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29b8111050 con 0x7f29b80fd130 2024-02-10T06:09:38.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.406+0000 7f29aeffd700 1 -- 172.21.15.173:0/91469994 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f29b401be90 con 0x7f29b80fd130 2024-02-10T06:09:38.407 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:38.407 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:38.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.408+0000 7f29c0466700 1 -- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f29a40368c0 msgr2=0x7f29a4038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:09:38.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.408+0000 7f29c0466700 1 --2- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f29a40368c0 0x7f29a4038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:38.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.408+0000 7f29c0466700 1 -- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29b80fd130 msgr2=0x7f29b810f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:38.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.408+0000 7f29c0466700 1 --2- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29b80fd130 0x7f29b810f120 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009330 tx=0x7f29b4009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:38.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.408+0000 7f29c0466700 1 -- 172.21.15.173:0/91469994 shutdown_connections 2024-02-10T06:09:38.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.408+0000 7f29c0466700 1 --2- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7f29a40368c0 0x7f29a4038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:38.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.408+0000 7f29c0466700 1 --2- 172.21.15.173:0/91469994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29b80fd130 0x7f29b810f120 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:38.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.408+0000 7f29c0466700 1 -- 172.21.15.173:0/91469994 >> 172.21.15.173:0/91469994 conn(0x7f29b80f69c0 msgr2=0x7f29b80f7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:38.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.408+0000 7f29c0466700 1 -- 172.21.15.173:0/91469994 shutdown_connections 2024-02-10T06:09:38.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:38.408+0000 7f29c0466700 1 -- 172.21.15.173:0/91469994 wait complete. 2024-02-10T06:09:38.410 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:39.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:38 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/91469994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:40.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:40.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:40.264 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:40.311 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:41.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.827+0000 7fc27aa11700 1 -- 172.21.15.173:0/3207813756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc274066880 msgr2=0x7fc2740fd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:41.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.827+0000 7fc27aa11700 1 --2- 172.21.15.173:0/3207813756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc274066880 0x7fc2740fd590 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc2680048c0 tx=0x7fc268009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:41.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.827+0000 7fc27aa11700 1 -- 172.21.15.173:0/3207813756 shutdown_connections 2024-02-10T06:09:41.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.827+0000 7fc27aa11700 1 --2- 172.21.15.173:0/3207813756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc274066880 0x7fc2740fd590 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.827+0000 7fc27aa11700 1 -- 172.21.15.173:0/3207813756 >> 172.21.15.173:0/3207813756 conn(0x7fc2740f7210 msgr2=0x7fc2740f9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.827+0000 7fc27aa11700 1 -- 172.21.15.173:0/3207813756 shutdown_connections 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.827+0000 7fc27aa11700 1 -- 172.21.15.173:0/3207813756 wait complete. 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.827+0000 7fc27aa11700 1 Processor -- start 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.828+0000 7fc27aa11700 1 -- start start 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.828+0000 7fc27aa11700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc274066880 0x7fc2741134d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.828+0000 7fc27aa11700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc274113a10 con 0x7fc274066880 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.828+0000 7fc273fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc274066880 0x7fc2741134d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.828+0000 7fc273fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc274066880 0x7fc2741134d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:55186/0 (socket says 172.21.15.173:55186) 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.828+0000 7fc273fff700 1 -- 172.21.15.173:0/1987207728 learned_addr learned my addr 172.21.15.173:0/1987207728 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.828+0000 7fc273fff700 1 -- 172.21.15.173:0/1987207728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc268005040 con 0x7fc274066880 2024-02-10T06:09:41.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.828+0000 7fc273fff700 1 --2- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc274066880 0x7fc2741134d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc268009b80 tx=0x7fc268009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:41.831 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.828+0000 7fc270ff9700 1 -- 172.21.15.173:0/1987207728 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc26800b5c0 con 0x7fc274066880 2024-02-10T06:09:41.831 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.829+0000 7fc27aa11700 1 -- 172.21.15.173:0/1987207728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc274113c10 con 0x7fc274066880 2024-02-10T06:09:41.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.829+0000 7fc270ff9700 1 -- 172.21.15.173:0/1987207728 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc26800be70 con 0x7fc274066880 2024-02-10T06:09:41.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.829+0000 7fc270ff9700 1 -- 172.21.15.173:0/1987207728 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc26801c660 con 0x7fc274066880 2024-02-10T06:09:41.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.829+0000 7fc27aa11700 1 -- 172.21.15.173:0/1987207728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2741148b0 con 0x7fc274066880 2024-02-10T06:09:41.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.829+0000 7fc270ff9700 1 -- 172.21.15.173:0/1987207728 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7fc26801c7c0 con 0x7fc274066880 2024-02-10T06:09:41.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.829+0000 7fc27aa11700 1 -- 172.21.15.173:0/1987207728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27405f5c0 con 0x7fc274066880 2024-02-10T06:09:41.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.829+0000 7fc270ff9700 1 --2- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc25c036910 0x7fc25c038dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:41.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.830+0000 7fc270ff9700 1 -- 172.21.15.173:0/1987207728 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc268049ac0 con 0x7fc274066880 2024-02-10T06:09:41.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.830+0000 7fc2737fe700 1 -- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc25c036910 msgr2=0x7fc25c038dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1955086590 2024-02-10T06:09:41.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.830+0000 7fc2737fe700 1 --2- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc25c036910 0x7fc25c038dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:09:41.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:41.834+0000 7fc270ff9700 1 -- 172.21.15.173:0/1987207728 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc268011dd0 con 0x7fc274066880 2024-02-10T06:09:42.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.031+0000 7fc2737fe700 1 -- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc25c036910 msgr2=0x7fc25c038dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/1955086590 2024-02-10T06:09:42.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.031+0000 7fc2737fe700 1 --2- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc25c036910 0x7fc25c038dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T06:09:42.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.090+0000 7fc270ff9700 1 -- 172.21.15.173:0/1987207728 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 15) v1 ==== 43162+0+0 (secure 0 0 0) 0x7fc268010d90 con 0x7fc274066880 2024-02-10T06:09:42.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.090+0000 7fc270ff9700 1 -- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc25c036910 msgr2=0x7fc25c038dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:09:42.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.090+0000 7fc270ff9700 1 --2- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc25c036910 0x7fc25c038dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:42.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.095+0000 7fc27aa11700 1 -- 172.21.15.173:0/1987207728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc274115400 con 0x7fc274066880 2024-02-10T06:09:42.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.095+0000 7fc270ff9700 1 -- 172.21.15.173:0/1987207728 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc268011dd0 con 0x7fc274066880 2024-02-10T06:09:42.096 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:42.096 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:42.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.097+0000 7fc27aa11700 1 -- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc274066880 msgr2=0x7fc2741134d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:42.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.097+0000 7fc27aa11700 1 --2- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc274066880 0x7fc2741134d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc268009b80 tx=0x7fc268009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:42.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.097+0000 7fc27aa11700 1 -- 172.21.15.173:0/1987207728 shutdown_connections 2024-02-10T06:09:42.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.097+0000 7fc27aa11700 1 --2- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:6800/1955086590,v1:172.21.15.89:6801/1955086590] conn(0x7fc25c036910 0x7fc25c038dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:42.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.097+0000 7fc27aa11700 1 --2- 172.21.15.173:0/1987207728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc274066880 0x7fc2741134d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:42.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.097+0000 7fc27aa11700 1 -- 172.21.15.173:0/1987207728 >> 172.21.15.173:0/1987207728 conn(0x7fc2740f7210 msgr2=0x7fc2740f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:42.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.097+0000 7fc27aa11700 1 -- 172.21.15.173:0/1987207728 shutdown_connections 2024-02-10T06:09:42.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:42.097+0000 7fc27aa11700 1 -- 172.21.15.173:0/1987207728 wait complete. 2024-02-10T06:09:42.099 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: Active manager daemon smithi089.pwmrye restarted 2024-02-10T06:09:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: Activating manager daemon smithi089.pwmrye 2024-02-10T06:09:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T06:09:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: mgrmap e15: smithi089.pwmrye(active, starting, since 0.00328724s) 2024-02-10T06:09:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-02-10T06:09:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr metadata", "who": "smithi089.pwmrye", "id": "smithi089.pwmrye"}]: dispatch 2024-02-10T06:09:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:09:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:09:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:09:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/1987207728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:42.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: Manager daemon smithi089.pwmrye is now available 2024-02-10T06:09:42.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:09:42.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:09:43.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:43 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:43 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:43 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.pwmrye/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:09:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:43 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.pwmrye/trash_purge_schedule"}]: dispatch 2024-02-10T06:09:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:43 smithi089 ceph-mon[29217]: [10/Feb/2024:06:09:42] ENGINE Bus STARTING 2024-02-10T06:09:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:43 smithi089 ceph-mon[29217]: [10/Feb/2024:06:09:42] ENGINE Serving on https://172.21.15.89:7150 2024-02-10T06:09:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:43 smithi089 ceph-mon[29217]: [10/Feb/2024:06:09:42] ENGINE Bus STARTED 2024-02-10T06:09:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:43 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:43 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:43 smithi089 ceph-mon[29217]: mgrmap e16: smithi089.pwmrye(active, since 1.00535s) 2024-02-10T06:09:43.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:43.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:44.015 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:44.065 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T06:09:44.489 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:44 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:44.489 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:44 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:45.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:45 smithi089 ceph-mon[29217]: mgrmap e17: smithi089.pwmrye(active, since 2s) 2024-02-10T06:09:46.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.736+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1422674777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f280fd710 msgr2=0x7f8f280ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:46.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.736+0000 7f8f2edc4700 1 --2- 172.21.15.173:0/1422674777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f280fd710 0x7f8f280ffaf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8f20009c60 tx=0x7f8f20004d10 comp rx=0 tx=0).stop 2024-02-10T06:09:46.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.736+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1422674777 shutdown_connections 2024-02-10T06:09:46.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.736+0000 7f8f2edc4700 1 --2- 172.21.15.173:0/1422674777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f280fd710 0x7f8f280ffaf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:46.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.736+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1422674777 >> 172.21.15.173:0/1422674777 conn(0x7f8f280f7190 msgr2=0x7f8f280f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:46.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.737+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1422674777 shutdown_connections 2024-02-10T06:09:46.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.737+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1422674777 wait complete. 2024-02-10T06:09:46.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.737+0000 7f8f2edc4700 1 Processor -- start 2024-02-10T06:09:46.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.737+0000 7f8f2edc4700 1 -- start start 2024-02-10T06:09:46.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.737+0000 7f8f2edc4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f280fd710 0x7f8f2810cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:46.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.737+0000 7f8f2edc4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2810d3e0 con 0x7f8f280fd710 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.738+0000 7f8f2cb60700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f280fd710 0x7f8f2810cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.738+0000 7f8f2cb60700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f280fd710 0x7f8f2810cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:37686/0 (socket says 172.21.15.173:37686) 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.738+0000 7f8f2cb60700 1 -- 172.21.15.173:0/1034358011 learned_addr learned my addr 172.21.15.173:0/1034358011 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.738+0000 7f8f2cb60700 1 -- 172.21.15.173:0/1034358011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f20009940 con 0x7f8f280fd710 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.738+0000 7f8f2cb60700 1 --2- 172.21.15.173:0/1034358011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f280fd710 0x7f8f2810cea0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8f20004e60 tx=0x7f8f20004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.738+0000 7f8f257fa700 1 -- 172.21.15.173:0/1034358011 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f2001d440 con 0x7f8f280fd710 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.738+0000 7f8f257fa700 1 -- 172.21.15.173:0/1034358011 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f2001da70 con 0x7f8f280fd710 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.738+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1034358011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f2810d5e0 con 0x7f8f280fd710 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.738+0000 7f8f257fa700 1 -- 172.21.15.173:0/1034358011 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f20012a30 con 0x7f8f280fd710 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.738+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1034358011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f2810e280 con 0x7f8f280fd710 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.739+0000 7f8f257fa700 1 -- 172.21.15.173:0/1034358011 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f8f20019040 con 0x7f8f280fd710 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.739+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1034358011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f2805f5c0 con 0x7f8f280fd710 2024-02-10T06:09:46.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.739+0000 7f8f257fa700 1 --2- 172.21.15.173:0/1034358011 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f8f10036960 0x7f8f10038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:46.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.739+0000 7f8f257fa700 1 -- 172.21.15.173:0/1034358011 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f8f2004a680 con 0x7f8f280fd710 2024-02-10T06:09:46.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.740+0000 7f8f27fff700 1 --2- 172.21.15.173:0/1034358011 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f8f10036960 0x7f8f10038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:46.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.740+0000 7f8f27fff700 1 --2- 172.21.15.173:0/1034358011 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f8f10036960 0x7f8f10038e10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009990 tx=0x7f8f1c006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:46.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:46.744+0000 7f8f257fa700 1 -- 172.21.15.173:0/1034358011 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8f20024310 con 0x7f8f280fd710 2024-02-10T06:09:47.014 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.012+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1034358011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f2804bed0 con 0x7f8f280fd710 2024-02-10T06:09:47.014 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.012+0000 7f8f257fa700 1 -- 172.21.15.173:0/1034358011 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f8f20017070 con 0x7f8f280fd710 2024-02-10T06:09:47.014 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:47.015 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:47.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.015+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1034358011 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f8f10036960 msgr2=0x7f8f10038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:47.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.015+0000 7f8f2edc4700 1 --2- 172.21.15.173:0/1034358011 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f8f10036960 0x7f8f10038e10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009990 tx=0x7f8f1c006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:47.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.015+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1034358011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f280fd710 msgr2=0x7f8f2810cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:47.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.015+0000 7f8f2edc4700 1 --2- 172.21.15.173:0/1034358011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f280fd710 0x7f8f2810cea0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8f20004e60 tx=0x7f8f20004d10 comp rx=0 tx=0).stop 2024-02-10T06:09:47.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.015+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1034358011 shutdown_connections 2024-02-10T06:09:47.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.015+0000 7f8f2edc4700 1 --2- 172.21.15.173:0/1034358011 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f8f10036960 0x7f8f10038e10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:47.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.015+0000 7f8f2edc4700 1 --2- 172.21.15.173:0/1034358011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f280fd710 0x7f8f2810cea0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:47.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.015+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1034358011 >> 172.21.15.173:0/1034358011 conn(0x7f8f280f7190 msgr2=0x7f8f280f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:47.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.015+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1034358011 shutdown_connections 2024-02-10T06:09:47.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:47.015+0000 7f8f2edc4700 1 -- 172.21.15.173:0/1034358011 wait complete. 2024-02-10T06:09:47.017 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:47.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:47 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:47.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:47 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:47.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:47 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:47.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:47 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:47.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:47 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd/host:smithi089", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:47.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:47 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/1034358011' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:48.669 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:48.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:48.919 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:09:49.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: Updating smithi173:/etc/ceph/ceph.conf 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: Updating smithi173:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: Updating smithi089:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:49.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:49.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:49.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:49.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi173", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:09:49.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi173", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T06:09:49.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:09:50.366 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.365+0000 7f7dd6473700 1 -- 172.21.15.173:0/2183876167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dd0102bb0 msgr2=0x7f7dd0102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:50.366 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.365+0000 7f7dd6473700 1 --2- 172.21.15.173:0/2183876167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dd0102bb0 0x7f7dd0102f80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7dc00048c0 tx=0x7f7dc0009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:50.366 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.365+0000 7f7dd6473700 1 -- 172.21.15.173:0/2183876167 shutdown_connections 2024-02-10T06:09:50.367 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.365+0000 7f7dd6473700 1 --2- 172.21.15.173:0/2183876167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dd0102bb0 0x7f7dd0102f80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:50.367 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.365+0000 7f7dd6473700 1 -- 172.21.15.173:0/2183876167 >> 172.21.15.173:0/2183876167 conn(0x7f7dd00f8760 msgr2=0x7f7dd00fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:50.367 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.365+0000 7f7dd6473700 1 -- 172.21.15.173:0/2183876167 shutdown_connections 2024-02-10T06:09:50.367 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.365+0000 7f7dd6473700 1 -- 172.21.15.173:0/2183876167 wait complete. 2024-02-10T06:09:50.367 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.366+0000 7f7dd6473700 1 Processor -- start 2024-02-10T06:09:50.367 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.366+0000 7f7dd6473700 1 -- start start 2024-02-10T06:09:50.367 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.366+0000 7f7dd6473700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dd0102bb0 0x7f7dd0112ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:50.367 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.366+0000 7f7dd6473700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd0113430 con 0x7f7dd0102bb0 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.367+0000 7f7dcffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dd0102bb0 0x7f7dd0112ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.367+0000 7f7dcffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dd0102bb0 0x7f7dd0112ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:37708/0 (socket says 172.21.15.173:37708) 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.367+0000 7f7dcffff700 1 -- 172.21.15.173:0/1097046183 learned_addr learned my addr 172.21.15.173:0/1097046183 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.367+0000 7f7dcffff700 1 -- 172.21.15.173:0/1097046183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc0005040 con 0x7f7dd0102bb0 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.367+0000 7f7dcffff700 1 --2- 172.21.15.173:0/1097046183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dd0102bb0 0x7f7dd0112ef0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0009330 tx=0x7f7dc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.367+0000 7f7dccff9700 1 -- 172.21.15.173:0/1097046183 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dc000be50 con 0x7f7dd0102bb0 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.367+0000 7f7dd6473700 1 -- 172.21.15.173:0/1097046183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd0113630 con 0x7f7dd0102bb0 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.367+0000 7f7dccff9700 1 -- 172.21.15.173:0/1097046183 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7dc0009630 con 0x7f7dd0102bb0 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.368+0000 7f7dccff9700 1 -- 172.21.15.173:0/1097046183 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dc001c3f0 con 0x7f7dd0102bb0 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.368+0000 7f7dd6473700 1 -- 172.21.15.173:0/1097046183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd0113b20 con 0x7f7dd0102bb0 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.368+0000 7f7dccff9700 1 -- 172.21.15.173:0/1097046183 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f7dc0013420 con 0x7f7dd0102bb0 2024-02-10T06:09:50.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.368+0000 7f7dd6473700 1 -- 172.21.15.173:0/1097046183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd005f5c0 con 0x7f7dd0102bb0 2024-02-10T06:09:50.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.369+0000 7f7dccff9700 1 --2- 172.21.15.173:0/1097046183 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7db8036960 0x7f7db8038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:50.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.369+0000 7f7dccff9700 1 -- 172.21.15.173:0/1097046183 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7dc0049c20 con 0x7f7dd0102bb0 2024-02-10T06:09:50.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.369+0000 7f7dcf7fe700 1 --2- 172.21.15.173:0/1097046183 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7db8036960 0x7f7db8038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:50.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.369+0000 7f7dcf7fe700 1 --2- 172.21.15.173:0/1097046183 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7db8036960 0x7f7db8038e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4009990 tx=0x7f7dc4006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:50.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.375+0000 7f7dccff9700 1 -- 172.21.15.173:0/1097046183 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7dc0011550 con 0x7f7dd0102bb0 2024-02-10T06:09:50.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:50 smithi089 ceph-mon[29217]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:09:50.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:50 smithi089 ceph-mon[29217]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:09:50.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:50 smithi089 ceph-mon[29217]: Updating smithi173:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:09:50.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:50 smithi089 ceph-mon[29217]: Updating smithi089:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:09:50.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:50 smithi089 ceph-mon[29217]: Deploying daemon crash.smithi173 on smithi173 2024-02-10T06:09:50.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.643+0000 7f7dd6473700 1 -- 172.21.15.173:0/1097046183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7dd010e540 con 0x7f7dd0102bb0 2024-02-10T06:09:50.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.644+0000 7f7dccff9700 1 -- 172.21.15.173:0/1097046183 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7dc0011370 con 0x7f7dd0102bb0 2024-02-10T06:09:50.644 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:50.645 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:50.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.646+0000 7f7dd6473700 1 -- 172.21.15.173:0/1097046183 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7db8036960 msgr2=0x7f7db8038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:50.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.646+0000 7f7dd6473700 1 --2- 172.21.15.173:0/1097046183 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7db8036960 0x7f7db8038e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4009990 tx=0x7f7dc4006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:50.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.646+0000 7f7dd6473700 1 -- 172.21.15.173:0/1097046183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dd0102bb0 msgr2=0x7f7dd0112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:50.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.646+0000 7f7dd6473700 1 --2- 172.21.15.173:0/1097046183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dd0102bb0 0x7f7dd0112ef0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0009330 tx=0x7f7dc0009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:50.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.646+0000 7f7dd6473700 1 -- 172.21.15.173:0/1097046183 shutdown_connections 2024-02-10T06:09:50.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.646+0000 7f7dd6473700 1 --2- 172.21.15.173:0/1097046183 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7db8036960 0x7f7db8038e10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:50.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.646+0000 7f7dd6473700 1 --2- 172.21.15.173:0/1097046183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dd0102bb0 0x7f7dd0112ef0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:50.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.646+0000 7f7dd6473700 1 -- 172.21.15.173:0/1097046183 >> 172.21.15.173:0/1097046183 conn(0x7f7dd00f8760 msgr2=0x7f7dd00fa650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:50.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.646+0000 7f7dd6473700 1 -- 172.21.15.173:0/1097046183 shutdown_connections 2024-02-10T06:09:50.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:50.646+0000 7f7dd6473700 1 -- 172.21.15.173:0/1097046183 wait complete. 2024-02-10T06:09:50.647 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:51.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:51 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/1097046183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:52.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:52.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:52.797 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:09:54.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.236+0000 7f19f8333700 1 -- 172.21.15.173:0/1758633670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f0102bb0 msgr2=0x7f19f0102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.236+0000 7f19f8333700 1 --2- 172.21.15.173:0/1758633670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f0102bb0 0x7f19f0102f80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f19ec0048c0 tx=0x7f19ec009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.236+0000 7f19f8333700 1 -- 172.21.15.173:0/1758633670 shutdown_connections 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.236+0000 7f19f8333700 1 --2- 172.21.15.173:0/1758633670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f0102bb0 0x7f19f0102f80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.236+0000 7f19f8333700 1 -- 172.21.15.173:0/1758633670 >> 172.21.15.173:0/1758633670 conn(0x7f19f00f8760 msgr2=0x7f19f00fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.236+0000 7f19f8333700 1 -- 172.21.15.173:0/1758633670 shutdown_connections 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.236+0000 7f19f8333700 1 -- 172.21.15.173:0/1758633670 wait complete. 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.237+0000 7f19f8333700 1 Processor -- start 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.237+0000 7f19f8333700 1 -- start start 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.237+0000 7f19f8333700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f0102bb0 0x7f19f0115120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.237+0000 7f19f8333700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f0115660 con 0x7f19f0102bb0 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.237+0000 7f19f60cf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f0102bb0 0x7f19f0115120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.237+0000 7f19f60cf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f0102bb0 0x7f19f0115120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:37720/0 (socket says 172.21.15.173:37720) 2024-02-10T06:09:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.237+0000 7f19f60cf700 1 -- 172.21.15.173:0/877029351 learned_addr learned my addr 172.21.15.173:0/877029351 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:54.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.238+0000 7f19f60cf700 1 -- 172.21.15.173:0/877029351 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec005040 con 0x7f19f0102bb0 2024-02-10T06:09:54.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.238+0000 7f19f60cf700 1 --2- 172.21.15.173:0/877029351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f0102bb0 0x7f19f0115120 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f19ec009300 tx=0x7f19ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:54.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.238+0000 7f19e6ffd700 1 -- 172.21.15.173:0/877029351 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19ec004e60 con 0x7f19f0102bb0 2024-02-10T06:09:54.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.238+0000 7f19e6ffd700 1 -- 172.21.15.173:0/877029351 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19ec013420 con 0x7f19f0102bb0 2024-02-10T06:09:54.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.238+0000 7f19f8333700 1 -- 172.21.15.173:0/877029351 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f0115860 con 0x7f19f0102bb0 2024-02-10T06:09:54.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.238+0000 7f19e6ffd700 1 -- 172.21.15.173:0/877029351 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19ec01c5b0 con 0x7f19f0102bb0 2024-02-10T06:09:54.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.238+0000 7f19f8333700 1 -- 172.21.15.173:0/877029351 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f0115d50 con 0x7f19f0102bb0 2024-02-10T06:09:54.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.239+0000 7f19e6ffd700 1 -- 172.21.15.173:0/877029351 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f19ec01c7f0 con 0x7f19f0102bb0 2024-02-10T06:09:54.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.239+0000 7f19e6ffd700 1 --2- 172.21.15.173:0/877029351 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f19dc036960 0x7f19dc038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:54.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.239+0000 7f19e6ffd700 1 -- 172.21.15.173:0/877029351 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f19ec049c00 con 0x7f19f0102bb0 2024-02-10T06:09:54.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.239+0000 7f19f8333700 1 -- 172.21.15.173:0/877029351 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f005f5c0 con 0x7f19f0102bb0 2024-02-10T06:09:54.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.239+0000 7f19f58ce700 1 --2- 172.21.15.173:0/877029351 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f19dc036960 0x7f19dc038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:54.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.240+0000 7f19f58ce700 1 --2- 172.21.15.173:0/877029351 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f19dc036960 0x7f19dc038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009990 tx=0x7f19e0006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:54.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.244+0000 7f19e6ffd700 1 -- 172.21.15.173:0/877029351 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f19ec025920 con 0x7f19f0102bb0 2024-02-10T06:09:54.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.515+0000 7f19f8333700 1 -- 172.21.15.173:0/877029351 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19f00008d0 con 0x7f19f0102bb0 2024-02-10T06:09:54.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.516+0000 7f19e6ffd700 1 -- 172.21.15.173:0/877029351 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f19ec011b60 con 0x7f19f0102bb0 2024-02-10T06:09:54.517 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:54.517 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:54.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.518+0000 7f19f8333700 1 -- 172.21.15.173:0/877029351 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f19dc036960 msgr2=0x7f19dc038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:54.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.518+0000 7f19f8333700 1 --2- 172.21.15.173:0/877029351 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f19dc036960 0x7f19dc038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009990 tx=0x7f19e0006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:54.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.518+0000 7f19f8333700 1 -- 172.21.15.173:0/877029351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f0102bb0 msgr2=0x7f19f0115120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:54.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.518+0000 7f19f8333700 1 --2- 172.21.15.173:0/877029351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f0102bb0 0x7f19f0115120 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f19ec009300 tx=0x7f19ec009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:54.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.518+0000 7f19f8333700 1 -- 172.21.15.173:0/877029351 shutdown_connections 2024-02-10T06:09:54.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.518+0000 7f19f8333700 1 --2- 172.21.15.173:0/877029351 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f19dc036960 0x7f19dc038e10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:54.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.518+0000 7f19f8333700 1 --2- 172.21.15.173:0/877029351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19f0102bb0 0x7f19f0115120 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:54.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.518+0000 7f19f8333700 1 -- 172.21.15.173:0/877029351 >> 172.21.15.173:0/877029351 conn(0x7f19f00f8760 msgr2=0x7f19f0071c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:54.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.518+0000 7f19f8333700 1 -- 172.21.15.173:0/877029351 shutdown_connections 2024-02-10T06:09:54.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:54.518+0000 7f19f8333700 1 -- 172.21.15.173:0/877029351 wait complete. 2024-02-10T06:09:54.535 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:54.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:54 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/877029351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:09:56.054 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:55 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:56.054 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:55 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:56.054 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:55 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:56.054 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:55 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:56.054 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:55 smithi089 ceph-mon[29217]: Deploying daemon node-exporter.smithi173 on smithi173 2024-02-10T06:09:56.388 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:09:56.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:09:56.644 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:09:58.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:09:58.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:58.962 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.916+0000 7f430f94d700 1 -- 172.21.15.173:0/1756171259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4308102bb0 msgr2=0x7f4308102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:58.962 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.916+0000 7f430f94d700 1 --2- 172.21.15.173:0/1756171259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4308102bb0 0x7f4308102f80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f42f80048c0 tx=0x7f42f8009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:58.962 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.916+0000 7f430f94d700 1 -- 172.21.15.173:0/1756171259 shutdown_connections 2024-02-10T06:09:58.962 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.916+0000 7f430f94d700 1 --2- 172.21.15.173:0/1756171259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4308102bb0 0x7f4308102f80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:58.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.916+0000 7f430f94d700 1 -- 172.21.15.173:0/1756171259 >> 172.21.15.173:0/1756171259 conn(0x7f43080f8760 msgr2=0x7f43080fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:58.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.916+0000 7f430f94d700 1 -- 172.21.15.173:0/1756171259 shutdown_connections 2024-02-10T06:09:58.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.916+0000 7f430f94d700 1 -- 172.21.15.173:0/1756171259 wait complete. 2024-02-10T06:09:58.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.916+0000 7f430f94d700 1 Processor -- start 2024-02-10T06:09:58.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.916+0000 7f430f94d700 1 -- start start 2024-02-10T06:09:58.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.917+0000 7f430f94d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4308102bb0 0x7f4308112f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:58.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.917+0000 7f430f94d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4308113460 con 0x7f4308102bb0 2024-02-10T06:09:58.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.917+0000 7f430d6e9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4308102bb0 0x7f4308112f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:58.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.917+0000 7f430d6e9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4308102bb0 0x7f4308112f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:32908/0 (socket says 172.21.15.173:32908) 2024-02-10T06:09:58.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.917+0000 7f430d6e9700 1 -- 172.21.15.173:0/3763269132 learned_addr learned my addr 172.21.15.173:0/3763269132 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:09:58.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.917+0000 7f430d6e9700 1 -- 172.21.15.173:0/3763269132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f8005040 con 0x7f4308102bb0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.917+0000 7f430d6e9700 1 --2- 172.21.15.173:0/3763269132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4308102bb0 0x7f4308112f20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f42f8009300 tx=0x7f42f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.917+0000 7f42fdffb700 1 -- 172.21.15.173:0/3763269132 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42f8004e60 con 0x7f4308102bb0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.917+0000 7f430f94d700 1 -- 172.21.15.173:0/3763269132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4308113660 con 0x7f4308102bb0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.918+0000 7f42fdffb700 1 -- 172.21.15.173:0/3763269132 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42f8013420 con 0x7f4308102bb0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.918+0000 7f42fdffb700 1 -- 172.21.15.173:0/3763269132 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42f801c5b0 con 0x7f4308102bb0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.918+0000 7f430f94d700 1 -- 172.21.15.173:0/3763269132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4308113b50 con 0x7f4308102bb0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.919+0000 7f430f94d700 1 -- 172.21.15.173:0/3763269132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430805f5c0 con 0x7f4308102bb0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.919+0000 7f42fdffb700 1 -- 172.21.15.173:0/3763269132 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f42f8013a90 con 0x7f4308102bb0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.919+0000 7f42fdffb700 1 --2- 172.21.15.173:0/3763269132 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f42f4036960 0x7f42f4038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.919+0000 7f42fdffb700 1 -- 172.21.15.173:0/3763269132 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f42f8049b30 con 0x7f4308102bb0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.919+0000 7f430cee8700 1 --2- 172.21.15.173:0/3763269132 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f42f4036960 0x7f42f4038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.919+0000 7f430cee8700 1 --2- 172.21.15.173:0/3763269132 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f42f4036960 0x7f42f4038e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4304009990 tx=0x7f4304006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:09:58.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:58.924+0000 7f42fdffb700 1 -- 172.21.15.173:0/3763269132 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f42f8011960 con 0x7f4308102bb0 2024-02-10T06:09:59.185 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.184+0000 7f430f94d700 1 -- 172.21.15.173:0/3763269132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f430810e6d0 con 0x7f4308102bb0 2024-02-10T06:09:59.185 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.185+0000 7f42fdffb700 1 -- 172.21.15.173:0/3763269132 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f42f8011370 con 0x7f4308102bb0 2024-02-10T06:09:59.185 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:09:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:09:59.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.187+0000 7f430f94d700 1 -- 172.21.15.173:0/3763269132 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f42f4036960 msgr2=0x7f42f4038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:59.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.187+0000 7f430f94d700 1 --2- 172.21.15.173:0/3763269132 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f42f4036960 0x7f42f4038e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4304009990 tx=0x7f4304006e40 comp rx=0 tx=0).stop 2024-02-10T06:09:59.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.187+0000 7f430f94d700 1 -- 172.21.15.173:0/3763269132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4308102bb0 msgr2=0x7f4308112f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:09:59.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.187+0000 7f430f94d700 1 --2- 172.21.15.173:0/3763269132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4308102bb0 0x7f4308112f20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f42f8009300 tx=0x7f42f8009e30 comp rx=0 tx=0).stop 2024-02-10T06:09:59.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.187+0000 7f430f94d700 1 -- 172.21.15.173:0/3763269132 shutdown_connections 2024-02-10T06:09:59.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.187+0000 7f430f94d700 1 --2- 172.21.15.173:0/3763269132 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f42f4036960 0x7f42f4038e10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:59.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.187+0000 7f430f94d700 1 --2- 172.21.15.173:0/3763269132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4308102bb0 0x7f4308112f20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:09:59.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.187+0000 7f430f94d700 1 -- 172.21.15.173:0/3763269132 >> 172.21.15.173:0/3763269132 conn(0x7f43080f8760 msgr2=0x7f43080fa650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:09:59.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.187+0000 7f430f94d700 1 -- 172.21.15.173:0/3763269132 shutdown_connections 2024-02-10T06:09:59.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:09:59.187+0000 7f430f94d700 1 -- 172.21.15.173:0/3763269132 wait complete. 2024-02-10T06:09:59.188 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:09:59.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:09:59 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/3763269132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:00.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:00 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:00.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:00 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:00.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:00 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:00.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:00 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:00.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:00 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.swdirm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:10:00.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:00 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.swdirm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T06:10:00.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:00 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:10:00.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:00 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:00.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:00 smithi089 ceph-mon[29217]: Deploying daemon mgr.smithi173.swdirm on smithi173 2024-02-10T06:10:00.955 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:10:00.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:10:01.217 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:10:03.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.282+0000 7f6fd759e700 1 -- 172.21.15.173:0/4277468670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd0101960 msgr2=0x7f6fd0101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:03.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.282+0000 7f6fd759e700 1 --2- 172.21.15.173:0/4277468670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd0101960 0x7f6fd0101d30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6fc40048c0 tx=0x7f6fc4009e30 comp rx=0 tx=0).stop 2024-02-10T06:10:03.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.282+0000 7f6fd759e700 1 -- 172.21.15.173:0/4277468670 shutdown_connections 2024-02-10T06:10:03.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.282+0000 7f6fd759e700 1 --2- 172.21.15.173:0/4277468670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd0101960 0x7f6fd0101d30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:03.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.282+0000 7f6fd759e700 1 -- 172.21.15.173:0/4277468670 >> 172.21.15.173:0/4277468670 conn(0x7f6fd00f74a0 msgr2=0x7f6fd00f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:10:03.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.282+0000 7f6fd759e700 1 -- 172.21.15.173:0/4277468670 shutdown_connections 2024-02-10T06:10:03.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.282+0000 7f6fd759e700 1 -- 172.21.15.173:0/4277468670 wait complete. 2024-02-10T06:10:03.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.283+0000 7f6fd759e700 1 Processor -- start 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.283+0000 7f6fd759e700 1 -- start start 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.283+0000 7f6fd759e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd0101960 0x7f6fd0112db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.283+0000 7f6fd759e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd01132f0 con 0x7f6fd0101960 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.283+0000 7f6fd659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd0101960 0x7f6fd0112db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.283+0000 7f6fd659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd0101960 0x7f6fd0112db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:32966/0 (socket says 172.21.15.173:32966) 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.283+0000 7f6fd659c700 1 -- 172.21.15.173:0/516320410 learned_addr learned my addr 172.21.15.173:0/516320410 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.283+0000 7f6fd659c700 1 -- 172.21.15.173:0/516320410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4005040 con 0x7f6fd0101960 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.284+0000 7f6fd659c700 1 --2- 172.21.15.173:0/516320410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd0101960 0x7f6fd0112db0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009300 tx=0x7f6fc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.284+0000 7f6fc2ffd700 1 -- 172.21.15.173:0/516320410 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fc4004e60 con 0x7f6fd0101960 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.284+0000 7f6fd759e700 1 -- 172.21.15.173:0/516320410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd01134f0 con 0x7f6fd0101960 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.284+0000 7f6fc2ffd700 1 -- 172.21.15.173:0/516320410 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fc4013420 con 0x7f6fd0101960 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.284+0000 7f6fc2ffd700 1 -- 172.21.15.173:0/516320410 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fc401c5b0 con 0x7f6fd0101960 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.284+0000 7f6fd759e700 1 -- 172.21.15.173:0/516320410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd01139e0 con 0x7f6fd0101960 2024-02-10T06:10:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.285+0000 7f6fd759e700 1 -- 172.21.15.173:0/516320410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd005f5c0 con 0x7f6fd0101960 2024-02-10T06:10:03.296 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.285+0000 7f6fc2ffd700 1 -- 172.21.15.173:0/516320410 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f6fc401c7f0 con 0x7f6fd0101960 2024-02-10T06:10:03.296 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.285+0000 7f6fc2ffd700 1 --2- 172.21.15.173:0/516320410 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f6fbc036960 0x7f6fbc038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:03.296 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.285+0000 7f6fc2ffd700 1 -- 172.21.15.173:0/516320410 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6fc4049d70 con 0x7f6fd0101960 2024-02-10T06:10:03.296 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.285+0000 7f6fd5d9b700 1 --2- 172.21.15.173:0/516320410 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f6fbc036960 0x7f6fbc038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:03.296 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.286+0000 7f6fd5d9b700 1 --2- 172.21.15.173:0/516320410 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f6fbc036960 0x7f6fbc038e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009990 tx=0x7f6fc8006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:10:03.296 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.290+0000 7f6fc2ffd700 1 -- 172.21.15.173:0/516320410 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6fc4023300 con 0x7f6fd0101960 2024-02-10T06:10:03.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.581+0000 7f6fd759e700 1 -- 172.21.15.173:0/516320410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6fd010e520 con 0x7f6fd0101960 2024-02-10T06:10:03.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.582+0000 7f6fc2ffd700 1 -- 172.21.15.173:0/516320410 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6fc4011b60 con 0x7f6fd0101960 2024-02-10T06:10:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:10:03.583 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:10:03.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.584+0000 7f6fd759e700 1 -- 172.21.15.173:0/516320410 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f6fbc036960 msgr2=0x7f6fbc038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:03.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.584+0000 7f6fd759e700 1 --2- 172.21.15.173:0/516320410 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f6fbc036960 0x7f6fbc038e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009990 tx=0x7f6fc8006e40 comp rx=0 tx=0).stop 2024-02-10T06:10:03.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.584+0000 7f6fd759e700 1 -- 172.21.15.173:0/516320410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd0101960 msgr2=0x7f6fd0112db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:03.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.584+0000 7f6fd759e700 1 --2- 172.21.15.173:0/516320410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd0101960 0x7f6fd0112db0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009300 tx=0x7f6fc4009e30 comp rx=0 tx=0).stop 2024-02-10T06:10:03.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.584+0000 7f6fd759e700 1 -- 172.21.15.173:0/516320410 shutdown_connections 2024-02-10T06:10:03.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.584+0000 7f6fd759e700 1 --2- 172.21.15.173:0/516320410 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f6fbc036960 0x7f6fbc038e10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:03.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.584+0000 7f6fd759e700 1 --2- 172.21.15.173:0/516320410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fd0101960 0x7f6fd0112db0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:03.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.584+0000 7f6fd759e700 1 -- 172.21.15.173:0/516320410 >> 172.21.15.173:0/516320410 conn(0x7f6fd00f74a0 msgr2=0x7f6fd00661f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:10:03.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.584+0000 7f6fd759e700 1 -- 172.21.15.173:0/516320410 shutdown_connections 2024-02-10T06:10:03.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:03.584+0000 7f6fd759e700 1 -- 172.21.15.173:0/516320410 wait complete. 2024-02-10T06:10:03.586 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:10:03.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:03 smithi089 ceph-mon[29217]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:03.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:03.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:03.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:03.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:03.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:03.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:10:03.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:04.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:04 smithi089 ceph-mon[29217]: Deploying daemon mon.smithi173 on smithi173 2024-02-10T06:10:04.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:04 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/516320410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:05.553 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:05 smithi089 ceph-mon[29217]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:05.720 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:10:05.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:10:06.195 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi173/config 2024-02-10T06:10:07.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:07 smithi089 ceph-mon[29217]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:07.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.765+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/1818558089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c78101bd0 msgr2=0x7f1c78101fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:07.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.765+0000 7f1c7d8f3700 1 --2- 172.21.15.173:0/1818558089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c78101bd0 0x7f1c78101fa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1c680048c0 tx=0x7f1c68009e30 comp rx=0 tx=0).stop 2024-02-10T06:10:07.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.765+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/1818558089 shutdown_connections 2024-02-10T06:10:07.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.765+0000 7f1c7d8f3700 1 --2- 172.21.15.173:0/1818558089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c78101bd0 0x7f1c78101fa0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:07.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.765+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/1818558089 >> 172.21.15.173:0/1818558089 conn(0x7f1c780729f0 msgr2=0x7f1c78072df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:10:07.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.765+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/1818558089 shutdown_connections 2024-02-10T06:10:07.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.765+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/1818558089 wait complete. 2024-02-10T06:10:07.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.766+0000 7f1c7d8f3700 1 Processor -- start 2024-02-10T06:10:07.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.766+0000 7f1c7d8f3700 1 -- start start 2024-02-10T06:10:07.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.766+0000 7f1c7d8f3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c78101bd0 0x7f1c7810e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:07.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.766+0000 7f1c7d8f3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7810e660 con 0x7f1c78101bd0 2024-02-10T06:10:07.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.766+0000 7f1c76ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c78101bd0 0x7f1c7810e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:07.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.766+0000 7f1c76ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c78101bd0 0x7f1c7810e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.173:51698/0 (socket says 172.21.15.173:51698) 2024-02-10T06:10:07.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.766+0000 7f1c76ffd700 1 -- 172.21.15.173:0/554090022 learned_addr learned my addr 172.21.15.173:0/554090022 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:10:07.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.767+0000 7f1c76ffd700 1 -- 172.21.15.173:0/554090022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c68005040 con 0x7f1c78101bd0 2024-02-10T06:10:07.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.767+0000 7f1c76ffd700 1 --2- 172.21.15.173:0/554090022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c78101bd0 0x7f1c7810e120 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009300 tx=0x7f1c68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:10:07.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.767+0000 7f1c5f7fe700 1 -- 172.21.15.173:0/554090022 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c68004e60 con 0x7f1c78101bd0 2024-02-10T06:10:07.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.767+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/554090022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7810e860 con 0x7f1c78101bd0 2024-02-10T06:10:07.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.767+0000 7f1c5f7fe700 1 -- 172.21.15.173:0/554090022 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c68013420 con 0x7f1c78101bd0 2024-02-10T06:10:07.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.767+0000 7f1c5f7fe700 1 -- 172.21.15.173:0/554090022 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c6801c5b0 con 0x7f1c78101bd0 2024-02-10T06:10:07.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.767+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/554090022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7810ed50 con 0x7f1c78101bd0 2024-02-10T06:10:07.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.768+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/554090022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c78100ef0 con 0x7f1c78101bd0 2024-02-10T06:10:07.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.768+0000 7f1c5f7fe700 1 -- 172.21.15.173:0/554090022 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f1c6801c7f0 con 0x7f1c78101bd0 2024-02-10T06:10:07.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.769+0000 7f1c5f7fe700 1 --2- 172.21.15.173:0/554090022 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1c60036970 0x7f1c60038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:07.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.769+0000 7f1c5f7fe700 1 -- 172.21.15.173:0/554090022 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1c680239e0 con 0x7f1c78101bd0 2024-02-10T06:10:07.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.769+0000 7f1c767fc700 1 --2- 172.21.15.173:0/554090022 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1c60036970 0x7f1c60038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:07.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.770+0000 7f1c767fc700 1 --2- 172.21.15.173:0/554090022 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1c60036970 0x7f1c60038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009990 tx=0x7f1c6c006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:10:07.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:07.774+0000 7f1c5f7fe700 1 -- 172.21.15.173:0/554090022 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1c68011dd0 con 0x7f1c78101bd0 2024-02-10T06:10:08.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.055+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/554090022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c7804bf00 con 0x7f1c78101bd0 2024-02-10T06:10:08.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.056+0000 7f1c5f7fe700 1 -- 172.21.15.173:0/554090022 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1c68011370 con 0x7f1c78101bd0 2024-02-10T06:10:08.056 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:10:08.057 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:07:31.076212Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T06:10:08.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.058+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/554090022 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1c60036970 msgr2=0x7f1c60038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:08.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.058+0000 7f1c7d8f3700 1 --2- 172.21.15.173:0/554090022 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1c60036970 0x7f1c60038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009990 tx=0x7f1c6c006e40 comp rx=0 tx=0).stop 2024-02-10T06:10:08.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.058+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/554090022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c78101bd0 msgr2=0x7f1c7810e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:08.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.058+0000 7f1c7d8f3700 1 --2- 172.21.15.173:0/554090022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c78101bd0 0x7f1c7810e120 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009300 tx=0x7f1c68009e30 comp rx=0 tx=0).stop 2024-02-10T06:10:08.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.058+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/554090022 shutdown_connections 2024-02-10T06:10:08.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.058+0000 7f1c7d8f3700 1 --2- 172.21.15.173:0/554090022 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1c60036970 0x7f1c60038e20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:08.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.058+0000 7f1c7d8f3700 1 --2- 172.21.15.173:0/554090022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c78101bd0 0x7f1c7810e120 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:08.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.058+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/554090022 >> 172.21.15.173:0/554090022 conn(0x7f1c780729f0 msgr2=0x7f1c78103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:10:08.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.058+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/554090022 shutdown_connections 2024-02-10T06:10:08.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:08.058+0000 7f1c7d8f3700 1 -- 172.21.15.173:0/554090022 wait complete. 2024-02-10T06:10:08.060 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-10T06:10:08.185 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 podman[36216]: 2024-02-10 06:10:08.164703487 +0000 UTC m=+0.347304260 container start 08af8aefe37b9f780e9a78d04b73dff68b2675cdd31e40df9331e84aca5b1d02 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi173, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=quincy-72ce7c1, maintainer=Guillaume Abrioux ) 2024-02-10T06:10:08.185 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T06:10:08.185 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable), process ceph-mon, pid 2 2024-02-10T06:10:08.185 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pidfile_write: ignore empty --pid-file 2024-02-10T06:10:08.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:08 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:08 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/554090022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: load: jerasure load: lrc 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: RocksDB version: 6.15.5 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Compile date Feb 9 2024 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: DB SUMMARY 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: DB Session ID: YJ4L5329I0FGI9PALMWZ 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: CURRENT file: CURRENT 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi173/store.db dir, Total Num: 0, files: 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi173/store.db: 000004.log size: 511 ; 2024-02-10T06:10:08.624 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.error_if_exists: 0 2024-02-10T06:10:08.625 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.create_if_missing: 0 2024-02-10T06:10:08.625 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.paranoid_checks: 1 2024-02-10T06:10:08.625 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T06:10:08.625 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.env: 0x56518a2aaa80 2024-02-10T06:10:08.625 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.fs: Posix File System 2024-02-10T06:10:08.625 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.info_log: 0x56518ba89cc0 2024-02-10T06:10:08.625 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T06:10:08.625 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.statistics: (nil) 2024-02-10T06:10:08.625 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.use_fsync: 0 2024-02-10T06:10:08.625 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_log_file_size: 0 2024-02-10T06:10:08.625 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.allow_fallocate: 1 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.use_direct_reads: 0 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.db_log_dir: 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi173/store.db 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T06:10:08.626 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T06:10:08.627 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T06:10:08.627 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T06:10:08.627 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T06:10:08.628 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T06:10:08.628 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T06:10:08.628 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T06:10:08.628 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.write_buffer_manager: 0x56518c7feb40 2024-02-10T06:10:08.629 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T06:10:08.629 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T06:10:08.629 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T06:10:08.629 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T06:10:08.629 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T06:10:08.629 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T06:10:08.629 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T06:10:08.629 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T06:10:08.629 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T06:10:08.629 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.unordered_write: 0 2024-02-10T06:10:08.629 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.row_cache: None 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.wal_filter: None 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.preserve_deletes: 0 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.two_write_queues: 0 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.atomic_flush: 0 2024-02-10T06:10:08.630 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.log_readahead_size: 0 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_background_jobs: 2 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_background_compactions: -1 2024-02-10T06:10:08.631 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_subcompactions: 1 2024-02-10T06:10:08.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T06:10:08.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T06:10:08.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T06:10:08.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T06:10:08.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T06:10:08.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T06:10:08.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T06:10:08.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T06:10:08.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_open_files: -1 2024-02-10T06:10:08.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T06:10:08.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_background_flushes: -1 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Compression algorithms supported: 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: kZSTD supported: 0 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: kXpressCompression supported: 0 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: kZlibCompression supported: 1 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: kSnappyCompression supported: 1 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: kLZ4Compression supported: 1 2024-02-10T06:10:08.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: kBZip2Compression supported: 0 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi173/store.db/MANIFEST-000003 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.merge_operator: 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_filter: None 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56518badef58) 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: cache_index_and_filter_blocks: 1 2024-02-10T06:10:08.634 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T06:10:08.635 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T06:10:08.635 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: pin_top_level_index_and_filter: 1 2024-02-10T06:10:08.635 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: index_type: 0 2024-02-10T06:10:08.635 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: data_block_index_type: 0 2024-02-10T06:10:08.635 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: index_shortening: 1 2024-02-10T06:10:08.635 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T06:10:08.635 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: hash_index_allow_collision: 1 2024-02-10T06:10:08.635 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: checksum: 1 2024-02-10T06:10:08.635 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: no_block_cache: 0 2024-02-10T06:10:08.635 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: block_cache: 0x56518bb11090 2024-02-10T06:10:08.635 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: block_cache_name: BinnedLRUCache 2024-02-10T06:10:08.636 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: block_cache_options: 2024-02-10T06:10:08.636 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: capacity : 536870912 2024-02-10T06:10:08.636 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: num_shard_bits : 4 2024-02-10T06:10:08.636 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: strict_capacity_limit : 0 2024-02-10T06:10:08.636 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: high_pri_pool_ratio: 0.000 2024-02-10T06:10:08.636 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: block_cache_compressed: (nil) 2024-02-10T06:10:08.636 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: persistent_cache: (nil) 2024-02-10T06:10:08.636 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: block_size: 4096 2024-02-10T06:10:08.636 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: block_size_deviation: 10 2024-02-10T06:10:08.636 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: block_restart_interval: 16 2024-02-10T06:10:08.636 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: index_block_restart_interval: 1 2024-02-10T06:10:08.637 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: metadata_block_size: 4096 2024-02-10T06:10:08.637 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: partition_filters: 0 2024-02-10T06:10:08.637 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: use_delta_encoding: 1 2024-02-10T06:10:08.637 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T06:10:08.637 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: whole_key_filtering: 1 2024-02-10T06:10:08.637 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: verify_compression: 0 2024-02-10T06:10:08.637 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: read_amp_bytes_per_bit: 0 2024-02-10T06:10:08.637 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: format_version: 4 2024-02-10T06:10:08.637 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: enable_index_compression: 1 2024-02-10T06:10:08.637 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: block_align: 0 2024-02-10T06:10:08.637 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T06:10:08.638 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T06:10:08.638 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compression: NoCompression 2024-02-10T06:10:08.638 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T06:10:08.638 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T06:10:08.638 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T06:10:08.638 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.num_levels: 7 2024-02-10T06:10:08.638 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T06:10:08.638 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T06:10:08.638 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T06:10:08.638 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T06:10:08.638 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T06:10:08.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T06:10:08.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T06:10:08.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T06:10:08.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T06:10:08.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T06:10:08.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T06:10:08.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T06:10:08.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T06:10:08.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T06:10:08.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T06:10:08.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T06:10:08.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T06:10:08.641 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T06:10:08.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T06:10:08.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T06:10:08.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T06:10:08.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T06:10:08.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T06:10:08.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T06:10:08.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T06:10:08.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T06:10:08.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.table_properties_collectors: 2024-02-10T06:10:08.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.inplace_update_support: 0 2024-02-10T06:10:08.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.bloom_locality: 0 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.max_successive_merges: 0 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.ttl: 2592000 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T06:10:08.643 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.enable_blob_files: false 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.min_blob_size: 0 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi173/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707545408191060, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707545408191356, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707545408, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "c8d41993-205d-41d0-8d46-00b775a46246", "db_session_id": "YJ4L5329I0FGI9PALMWZ"}} 2024-02-10T06:10:08.644 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707545408191654, "job": 1, "event": "recovery_finished"} 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi173/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x56518baa5180 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: DB pointer 0x56518bb7a000 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: ** DB Stats ** 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T06:10:08.645 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: ** Compaction Stats [default] ** 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: ** Compaction Stats [default] ** 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:10:08.646 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:10:08.647 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: ** Compaction Stats [default] ** 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: ** Compaction Stats [default] ** 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-10T06:10:08.648 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T06:10:08.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T06:10:08.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T06:10:08.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T06:10:08.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T06:10:08.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T06:10:08.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:10:08.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T06:10:08.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T06:10:08.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: 2024-02-10T06:10:08.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T06:10:08.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173 does not exist in monmap, will attempt to join an existing cluster 2024-02-10T06:10:08.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: using public_addr v2:172.21.15.173:0/0 -> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-02-10T06:10:08.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: starting mon.smithi173 rank -1 at public addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] at bind addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi173 fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:10:08.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(???) e0 preinit fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:10:08.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).mds e1 new map 2024-02-10T06:10:08.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).mds e1 print_map 2024-02-10T06:10:08.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: e1 2024-02-10T06:10:08.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:10:08.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:10:08.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: legacy client fscid: -1 2024-02-10T06:10:08.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: 2024-02-10T06:10:08.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout: No filesystems configured 2024-02-10T06:10:08.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T06:10:08.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T06:10:08.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T06:10:08.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T06:10:08.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T06:10:08.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T06:10:08.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-02-10T06:10:08.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T06:10:08.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:10:08.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Deploying cephadm binary to smithi173 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Added host smithi173 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Deploying daemon alertmanager.smithi089 on smithi089 2024-02-10T06:10:08.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/828227090' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/828227090' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.654 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Deploying daemon grafana.smithi089 on smithi089 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi089:172.21.15.89=smithi089;smithi173:172.21.15.173=smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Saving service mon spec with placement smithi089:172.21.15.89=smithi089;smithi173:172.21.15.173=smithi173;count:2 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.655 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/3529416868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/3754811506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/168033500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.656 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Deploying daemon prometheus.smithi089 on smithi089 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/1017078856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/3074145562' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/1175970889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.657 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/176553744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14162 172.21.15.89:0/589444550' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mgrmap e14: smithi089.pwmrye(active, since 57s) 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/91469994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Active manager daemon smithi089.pwmrye restarted 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Activating manager daemon smithi089.pwmrye 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T06:10:08.658 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mgrmap e15: smithi089.pwmrye(active, starting, since 0.00328724s) 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr metadata", "who": "smithi089.pwmrye", "id": "smithi089.pwmrye"}]: dispatch 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/1987207728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Manager daemon smithi089.pwmrye is now available 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:08.659 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.pwmrye/mirror_snapshot_schedule"}]: dispatch 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi089.pwmrye/trash_purge_schedule"}]: dispatch 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: [10/Feb/2024:06:09:42] ENGINE Bus STARTING 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: [10/Feb/2024:06:09:42] ENGINE Serving on https://172.21.15.89:7150 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: [10/Feb/2024:06:09:42] ENGINE Bus STARTED 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mgrmap e16: smithi089.pwmrye(active, since 1.00535s) 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mgrmap e17: smithi089.pwmrye(active, since 2s) 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd/host:smithi089", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/1034358011' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-10T06:10:08.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Updating smithi173:/etc/ceph/ceph.conf 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Updating smithi173:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Updating smithi089:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi173", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi173", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:10:08.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Updating smithi173:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Updating smithi089:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Deploying daemon crash.smithi173 on smithi173 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/1097046183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/877029351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Deploying daemon node-exporter.smithi173 on smithi173 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/3763269132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.664 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.664 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.664 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.664 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.swdirm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:10:08.664 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.swdirm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T06:10:08.664 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:10:08.664 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:08.664 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Deploying daemon mgr.smithi173.swdirm on smithi173 2024-02-10T06:10:08.664 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: Deploying daemon mon.smithi173 on smithi173 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/516320410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/554090022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:08.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-02-10T06:10:08.666 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 ceph-mon[36251]: mon.smithi173@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-10T06:10:08.666 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 bash[36216]: 08af8aefe37b9f780e9a78d04b73dff68b2675cdd31e40df9331e84aca5b1d02 2024-02-10T06:10:08.666 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:08 smithi173 systemd[1]: Started Ceph mon.smithi173 for 84e5b1ba-c7da-11ee-95b8-87774f69a715. 2024-02-10T06:10:09.915 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T06:10:09.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T06:10:10.163 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi173/config 2024-02-10T06:10:13.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:13.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-02-10T06:10:13.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: mon.smithi089 calling monitor election 2024-02-10T06:10:13.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.? 172.21.15.173:0/3175264405' entity='mgr.smithi173.swdirm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi173.swdirm/crt"}]: dispatch 2024-02-10T06:10:13.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:13.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: mon.smithi173 calling monitor election 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: mon.smithi089 is new leader, mons smithi089,smithi173 in quorum (ranks 0,1) 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: monmap e2: 2 mons at {smithi089=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],smithi173=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: fsmap 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: mgrmap e17: smithi089.pwmrye(active, since 31s) 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: Standby manager daemon smithi173.swdirm started 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.? 172.21.15.173:0/3175264405' entity='mgr.smithi173.swdirm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: overall HEALTH_OK 2024-02-10T06:10:13.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.? 172.21.15.173:0/3175264405' entity='mgr.smithi173.swdirm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi173.swdirm/key"}]: dispatch 2024-02-10T06:10:13.607 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.? 172.21.15.173:0/3175264405' entity='mgr.smithi173.swdirm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:10:13.607 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:13.607 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-02-10T06:10:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: mon.smithi089 calling monitor election 2024-02-10T06:10:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.? 172.21.15.173:0/3175264405' entity='mgr.smithi173.swdirm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi173.swdirm/crt"}]: dispatch 2024-02-10T06:10:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: mon.smithi173 calling monitor election 2024-02-10T06:10:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: mon.smithi089 is new leader, mons smithi089,smithi173 in quorum (ranks 0,1) 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: monmap e2: 2 mons at {smithi089=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0],smithi173=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: fsmap 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: mgrmap e17: smithi089.pwmrye(active, since 31s) 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: Standby manager daemon smithi173.swdirm started 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.? 172.21.15.173:0/3175264405' entity='mgr.smithi173.swdirm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: overall HEALTH_OK 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.? 172.21.15.173:0/3175264405' entity='mgr.smithi173.swdirm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi173.swdirm/key"}]: dispatch 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.? 172.21.15.173:0/3175264405' entity='mgr.smithi173.swdirm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:13.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:14.580 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:14 smithi173 ceph-mon[36251]: mgrmap e18: smithi089.pwmrye(active, since 31s), standbys: smithi173.swdirm 2024-02-10T06:10:14.580 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:14 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr metadata", "who": "smithi173.swdirm", "id": "smithi173.swdirm"}]: dispatch 2024-02-10T06:10:14.581 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:14 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:14.581 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:14 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:14.581 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:14 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:14.581 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:14 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:14.581 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:14 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:14.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:14 smithi089 ceph-mon[29217]: mgrmap e18: smithi089.pwmrye(active, since 31s), standbys: smithi173.swdirm 2024-02-10T06:10:14.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:14 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr metadata", "who": "smithi173.swdirm", "id": "smithi173.swdirm"}]: dispatch 2024-02-10T06:10:14.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:14 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:14.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:14 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:14.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:14 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:14.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:14 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:14.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:14 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:10:14.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.662+0000 7f0e7d952700 1 -- 172.21.15.173:0/4283273591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e64003580 msgr2=0x7f0e64005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:14.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.662+0000 7f0e7d952700 1 --2- 172.21.15.173:0/4283273591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e64003580 0x7f0e64005a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0e78065f20 tx=0x7f0e68009040 comp rx=0 tx=0).stop 2024-02-10T06:10:14.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.662+0000 7f0e7d952700 1 -- 172.21.15.173:0/4283273591 shutdown_connections 2024-02-10T06:10:14.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.662+0000 7f0e7d952700 1 --2- 172.21.15.173:0/4283273591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e64003580 0x7f0e64005a00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:14.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.662+0000 7f0e7d952700 1 -- 172.21.15.173:0/4283273591 >> 172.21.15.173:0/4283273591 conn(0x7f0e780f7740 msgr2=0x7f0e780f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:10:14.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.662+0000 7f0e7d952700 1 -- 172.21.15.173:0/4283273591 shutdown_connections 2024-02-10T06:10:14.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.662+0000 7f0e7d952700 1 -- 172.21.15.173:0/4283273591 wait complete. 2024-02-10T06:10:14.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.663+0000 7f0e7d952700 1 Processor -- start 2024-02-10T06:10:14.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.663+0000 7f0e7d952700 1 -- start start 2024-02-10T06:10:14.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e7d952700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e64003580 0x7f0e781131e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:14.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e7d952700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e78113720 0x7f0e7810d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e7d952700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e78113d30 con 0x7f0e64003580 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e7d952700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7810d5c0 con 0x7f0e78113720 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e767fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e78113720 0x7f0e7810d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e767fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e78113720 0x7f0e7810d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:43964/0 (socket says 172.21.15.173:43964) 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e767fc700 1 -- 172.21.15.173:0/3788944475 learned_addr learned my addr 172.21.15.173:0/3788944475 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e76ffd700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e64003580 0x7f0e781131e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e767fc700 1 -- 172.21.15.173:0/3788944475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e78113720 msgr2=0x7f0e7810d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e767fc700 1 -- 172.21.15.173:0/3788944475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e78113720 msgr2=0x7f0e7810d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e767fc700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e78113720 0x7f0e7810d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e767fc700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e78113720 0x7f0e7810d080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e76ffd700 1 -- 172.21.15.173:0/3788944475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e78113720 msgr2=0x7f0e7810d080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e76ffd700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e78113720 0x7f0e7810d080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:14.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e76ffd700 1 -- 172.21.15.173:0/3788944475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e680079e0 con 0x7f0e64003580 2024-02-10T06:10:14.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.664+0000 7f0e76ffd700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e64003580 0x7f0e781131e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0e680139e0 tx=0x7f0e68009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:10:14.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.665+0000 7f0e6f7fe700 1 -- 172.21.15.173:0/3788944475 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e68017750 con 0x7f0e64003580 2024-02-10T06:10:14.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.665+0000 7f0e6f7fe700 1 -- 172.21.15.173:0/3788944475 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e68017d80 con 0x7f0e64003580 2024-02-10T06:10:14.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.665+0000 7f0e7d952700 1 -- 172.21.15.173:0/3788944475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7810d840 con 0x7f0e64003580 2024-02-10T06:10:14.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.665+0000 7f0e6f7fe700 1 -- 172.21.15.173:0/3788944475 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e68020cf0 con 0x7f0e64003580 2024-02-10T06:10:14.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.665+0000 7f0e7d952700 1 -- 172.21.15.173:0/3788944475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7810dd80 con 0x7f0e64003580 2024-02-10T06:10:14.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.666+0000 7f0e7d952700 1 -- 172.21.15.173:0/3788944475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7805f5c0 con 0x7f0e64003580 2024-02-10T06:10:14.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.666+0000 7f0e6f7fe700 1 -- 172.21.15.173:0/3788944475 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f0e680178b0 con 0x7f0e64003580 2024-02-10T06:10:14.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.666+0000 7f0e6f7fe700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e58069320 0x7f0e5806b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:14.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.667+0000 7f0e6f7fe700 1 -- 172.21.15.173:0/3788944475 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0e68088960 con 0x7f0e64003580 2024-02-10T06:10:14.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.667+0000 7f0e767fc700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e58069320 0x7f0e5806b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:14.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.667+0000 7f0e767fc700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e58069320 0x7f0e5806b7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e60006fd0 tx=0x7f0e60008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:10:14.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.671+0000 7f0e6f7fe700 1 -- 172.21.15.173:0/3788944475 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0e680596e0 con 0x7f0e64003580 2024-02-10T06:10:14.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.937+0000 7f0e7d952700 1 -- 172.21.15.173:0/3788944475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e7810e7c0 con 0x7f0e64003580 2024-02-10T06:10:14.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.938+0000 7f0e6f7fe700 1 -- 172.21.15.173:0/3788944475 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7f0e6802f090 con 0x7f0e64003580 2024-02-10T06:10:14.938 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-10T06:10:14.938 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","modified":"2024-02-10T06:10:08.216768Z","created":"2024-02-10T06:07:31.076212Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi173","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T06:10:14.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.940+0000 7f0e7d952700 1 -- 172.21.15.173:0/3788944475 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e58069320 msgr2=0x7f0e5806b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.940+0000 7f0e7d952700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e58069320 0x7f0e5806b7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e60006fd0 tx=0x7f0e60008040 comp rx=0 tx=0).stop 2024-02-10T06:10:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.940+0000 7f0e7d952700 1 -- 172.21.15.173:0/3788944475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e64003580 msgr2=0x7f0e781131e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.940+0000 7f0e7d952700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e64003580 0x7f0e781131e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0e680139e0 tx=0x7f0e68009040 comp rx=0 tx=0).stop 2024-02-10T06:10:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.940+0000 7f0e7d952700 1 -- 172.21.15.173:0/3788944475 shutdown_connections 2024-02-10T06:10:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.940+0000 7f0e7d952700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e58069320 0x7f0e5806b7d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.940+0000 7f0e7d952700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e64003580 0x7f0e781131e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.940+0000 7f0e7d952700 1 --2- 172.21.15.173:0/3788944475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e78113720 0x7f0e7810d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.940+0000 7f0e7d952700 1 -- 172.21.15.173:0/3788944475 >> 172.21.15.173:0/3788944475 conn(0x7f0e780f7740 msgr2=0x7f0e780fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:10:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.940+0000 7f0e7d952700 1 -- 172.21.15.173:0/3788944475 shutdown_connections 2024-02-10T06:10:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:10:14.940+0000 7f0e7d952700 1 -- 172.21.15.173:0/3788944475 wait complete. 2024-02-10T06:10:14.942 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-02-10T06:10:15.503 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:15 smithi173 ceph-mon[36251]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:15.503 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:15 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/3788944475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:15.526 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T06:10:15.526 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T06:10:15.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:15 smithi089 ceph-mon[29217]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:15.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:15 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/3788944475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T06:10:15.798 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:10:16.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:16.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:16.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:16.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:16.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-10T06:10:16.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: Updating smithi173:/etc/ceph/ceph.conf 2024-02-10T06:10:16.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: Updating smithi089:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:10:16.971 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: Updating smithi173:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:10:16.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:16.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:16.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:16.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:16.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:16.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:16.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: Reconfiguring mon.smithi089 (unknown last config time)... 2024-02-10T06:10:16.972 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:10:16.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:10:16.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:16.973 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:16 smithi089 ceph-mon[29217]: Reconfiguring daemon mon.smithi089 on smithi089 2024-02-10T06:10:17.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:17.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:17.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:17.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:17.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: Updating smithi173:/etc/ceph/ceph.conf 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: Updating smithi089:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: Updating smithi173:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: Reconfiguring mon.smithi089 (unknown last config time)... 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:17.151 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:16 smithi173 ceph-mon[36251]: Reconfiguring daemon mon.smithi089 on smithi089 2024-02-10T06:10:17.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.509+0000 7fd937c08700 1 -- 172.21.15.89:0/2848770558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd930101b00 msgr2=0x7fd930101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.509+0000 7fd937c08700 1 --2- 172.21.15.89:0/2848770558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd930101b00 0x7fd930101ed0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd92c0048c0 tx=0x7fd92c009e30 comp rx=0 tx=0).stop 2024-02-10T06:10:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.509+0000 7fd937c08700 1 -- 172.21.15.89:0/2848770558 shutdown_connections 2024-02-10T06:10:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.509+0000 7fd937c08700 1 --2- 172.21.15.89:0/2848770558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd930101b00 0x7fd930101ed0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.509+0000 7fd937c08700 1 -- 172.21.15.89:0/2848770558 >> 172.21.15.89:0/2848770558 conn(0x7fd9300f7640 msgr2=0x7fd9300f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:10:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.509+0000 7fd937c08700 1 -- 172.21.15.89:0/2848770558 shutdown_connections 2024-02-10T06:10:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.510+0000 7fd937c08700 1 -- 172.21.15.89:0/2848770558 wait complete. 2024-02-10T06:10:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.510+0000 7fd937c08700 1 Processor -- start 2024-02-10T06:10:17.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.510+0000 7fd937c08700 1 -- start start 2024-02-10T06:10:17.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.510+0000 7fd937c08700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd930101b00 0x7fd930111f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:17.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.510+0000 7fd937c08700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9301124b0 0x7fd93010be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:17.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.510+0000 7fd937c08700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd930112ac0 con 0x7fd930101b00 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.510+0000 7fd937c08700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93010c350 con 0x7fd9301124b0 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd9359a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd930101b00 0x7fd930111f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd9359a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd930101b00 0x7fd930111f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39494/0 (socket says 172.21.15.89:39494) 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd9359a4700 1 -- 172.21.15.89:0/2202579592 learned_addr learned my addr 172.21.15.89:0/2202579592 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd9359a4700 1 -- 172.21.15.89:0/2202579592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9301124b0 msgr2=0x7fd93010be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd9359a4700 1 --2- 172.21.15.89:0/2202579592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9301124b0 0x7fd93010be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd9359a4700 1 -- 172.21.15.89:0/2202579592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c005040 con 0x7fd930101b00 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd9359a4700 1 --2- 172.21.15.89:0/2202579592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd930101b00 0x7fd930111f70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00b870 tx=0x7fd92c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd9267fc700 1 -- 172.21.15.89:0/2202579592 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd92c00e040 con 0x7fd930101b00 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd937c08700 1 -- 172.21.15.89:0/2202579592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93010c5d0 con 0x7fd930101b00 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd9267fc700 1 -- 172.21.15.89:0/2202579592 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd92c013420 con 0x7fd930101b00 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd9267fc700 1 -- 172.21.15.89:0/2202579592 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd92c01c630 con 0x7fd930101b00 2024-02-10T06:10:17.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.511+0000 7fd937c08700 1 -- 172.21.15.89:0/2202579592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93010cb90 con 0x7fd930101b00 2024-02-10T06:10:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.512+0000 7fd937c08700 1 -- 172.21.15.89:0/2202579592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd930100d70 con 0x7fd930101b00 2024-02-10T06:10:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.512+0000 7fd9267fc700 1 -- 172.21.15.89:0/2202579592 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fd92c013a20 con 0x7fd930101b00 2024-02-10T06:10:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.513+0000 7fd9267fc700 1 --2- 172.21.15.89:0/2202579592 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd91c069370 0x7fd91c06b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.513+0000 7fd9267fc700 1 -- 172.21.15.89:0/2202579592 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd92c0873f0 con 0x7fd930101b00 2024-02-10T06:10:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.513+0000 7fd9351a3700 1 --2- 172.21.15.89:0/2202579592 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd91c069370 0x7fd91c06b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:17.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.513+0000 7fd9351a3700 1 --2- 172.21.15.89:0/2202579592 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd91c069370 0x7fd91c06b820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd920009930 tx=0x7fd920006de0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:10:17.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.517+0000 7fd9267fc700 1 -- 172.21.15.89:0/2202579592 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd92c058290 con 0x7fd930101b00 2024-02-10T06:10:17.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.708+0000 7fd937c08700 1 -- 172.21.15.89:0/2202579592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd93004bed0 con 0x7fd930101b00 2024-02-10T06:10:17.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.708+0000 7fd9267fc700 1 -- 172.21.15.89:0/2202579592 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7fd92c086ba0 con 0x7fd930101b00 2024-02-10T06:10:17.709 INFO:teuthology.orchestra.run.smithi089.stdout:# minimal ceph.conf for 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:10:17.709 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-02-10T06:10:17.710 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 84e5b1ba-c7da-11ee-95b8-87774f69a715 2024-02-10T06:10:17.710 INFO:teuthology.orchestra.run.smithi089.stdout: mon_host = [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-02-10T06:10:17.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.711+0000 7fd937c08700 1 -- 172.21.15.89:0/2202579592 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd91c069370 msgr2=0x7fd91c06b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:17.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.711+0000 7fd937c08700 1 --2- 172.21.15.89:0/2202579592 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd91c069370 0x7fd91c06b820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd920009930 tx=0x7fd920006de0 comp rx=0 tx=0).stop 2024-02-10T06:10:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.711+0000 7fd937c08700 1 -- 172.21.15.89:0/2202579592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd930101b00 msgr2=0x7fd930111f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.711+0000 7fd937c08700 1 --2- 172.21.15.89:0/2202579592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd930101b00 0x7fd930111f70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00b870 tx=0x7fd92c009e30 comp rx=0 tx=0).stop 2024-02-10T06:10:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.711+0000 7fd937c08700 1 -- 172.21.15.89:0/2202579592 shutdown_connections 2024-02-10T06:10:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.711+0000 7fd937c08700 1 --2- 172.21.15.89:0/2202579592 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd91c069370 0x7fd91c06b820 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.711+0000 7fd937c08700 1 --2- 172.21.15.89:0/2202579592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd930101b00 0x7fd930111f70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.711+0000 7fd937c08700 1 --2- 172.21.15.89:0/2202579592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9301124b0 0x7fd93010be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.711+0000 7fd937c08700 1 -- 172.21.15.89:0/2202579592 >> 172.21.15.89:0/2202579592 conn(0x7fd9300f7640 msgr2=0x7fd9300f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:10:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.711+0000 7fd937c08700 1 -- 172.21.15.89:0/2202579592 shutdown_connections 2024-02-10T06:10:17.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:17.711+0000 7fd937c08700 1 -- 172.21.15.89:0/2202579592 wait complete. 2024-02-10T06:10:18.476 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:18 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:18.477 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:18 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2202579592' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:18.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:18 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:18.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:18 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2202579592' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:18.968 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T06:10:18.968 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:10:18.968 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T06:10:19.000 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:10:19.000 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:10:19.068 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:10:19.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T06:10:19.099 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:10:19.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T06:10:19.165 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T06:10:19.166 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:10:19.166 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T06:10:19.181 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T06:10:19.181 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2024-02-10T06:10:19.236 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T06:10:19.236 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:10:19.236 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-10T06:10:19.236 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:10:19.236 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:10:19.236 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-02-10 06:09:33.907016618 +0000 2024-02-10T06:10:19.236 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-02-10 06:03:48.232859133 +0000 2024-02-10T06:10:19.237 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-02-10 06:03:48.232859133 +0000 2024-02-10T06:10:19.237 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-02-10 06:03:48.232859133 +0000 2024-02-10T06:10:19.237 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T06:10:19.301 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-02-10T06:10:19.301 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-02-10T06:10:19.301 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000190416 s, 2.7 MB/s 2024-02-10T06:10:19.302 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T06:10:19.359 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2024-02-10T06:10:19.414 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T06:10:19.414 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:10:19.414 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-10T06:10:19.414 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:10:19.414 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:10:19.414 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-02-10 06:09:33.907016618 +0000 2024-02-10T06:10:19.414 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-02-10 06:03:48.586851001 +0000 2024-02-10T06:10:19.414 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-02-10 06:03:48.586851001 +0000 2024-02-10T06:10:19.415 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-02-10 06:03:48.586851001 +0000 2024-02-10T06:10:19.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T06:10:19.480 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-02-10T06:10:19.481 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-02-10T06:10:19.481 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000213598 s, 2.4 MB/s 2024-02-10T06:10:19.482 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T06:10:19.538 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2024-02-10T06:10:19.593 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T06:10:19.593 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:10:19.593 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-10T06:10:19.593 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:10:19.593 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:10:19.593 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-02-10 06:09:33.908016595 +0000 2024-02-10T06:10:19.593 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-02-10 06:03:48.930843098 +0000 2024-02-10T06:10:19.593 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-02-10 06:03:48.930843098 +0000 2024-02-10T06:10:19.593 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-02-10 06:03:48.930843098 +0000 2024-02-10T06:10:19.594 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T06:10:19.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:19 smithi089 ceph-mon[29217]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:19.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:19.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:19.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:19 smithi089 ceph-mon[29217]: Reconfiguring mgr.smithi089.pwmrye (unknown last config time)... 2024-02-10T06:10:19.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi089.pwmrye", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:10:19.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:10:19.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:19.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:19 smithi089 ceph-mon[29217]: Reconfiguring daemon mgr.smithi089.pwmrye on smithi089 2024-02-10T06:10:19.629 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-02-10T06:10:19.630 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-02-10T06:10:19.630 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000192435 s, 2.7 MB/s 2024-02-10T06:10:19.631 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T06:10:19.687 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2024-02-10T06:10:19.742 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T06:10:19.742 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:10:19.742 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-10T06:10:19.742 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:10:19.742 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:10:19.742 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-02-10 06:09:33.908016595 +0000 2024-02-10T06:10:19.742 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-02-10 06:03:49.248835792 +0000 2024-02-10T06:10:19.743 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-02-10 06:03:49.248835792 +0000 2024-02-10T06:10:19.743 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-02-10 06:03:49.248835792 +0000 2024-02-10T06:10:19.743 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T06:10:19.809 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-02-10T06:10:19.809 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-02-10T06:10:19.809 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000206235 s, 2.5 MB/s 2024-02-10T06:10:19.810 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T06:10:19.835 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:19 smithi173 ceph-mon[36251]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:19.836 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:19.836 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:19.836 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:19 smithi173 ceph-mon[36251]: Reconfiguring mgr.smithi089.pwmrye (unknown last config time)... 2024-02-10T06:10:19.836 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi089.pwmrye", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:10:19.836 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:10:19.836 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:19.836 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:19 smithi173 ceph-mon[36251]: Reconfiguring daemon mgr.smithi089.pwmrye on smithi089 2024-02-10T06:10:19.866 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:10:19.866 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T06:10:19.883 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T06:10:19.883 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-02-10T06:10:19.938 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T06:10:19.938 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:10:19.938 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-10T06:10:19.938 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:10:19.938 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:10:19.938 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-02-10 06:10:06.410286983 +0000 2024-02-10T06:10:19.938 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-02-10 06:03:48.237812446 +0000 2024-02-10T06:10:19.938 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-02-10 06:03:48.237812446 +0000 2024-02-10T06:10:19.938 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-02-10 06:03:48.237812446 +0000 2024-02-10T06:10:19.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T06:10:20.006 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-02-10T06:10:20.006 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-02-10T06:10:20.006 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000195056 s, 2.6 MB/s 2024-02-10T06:10:20.008 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T06:10:20.063 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-02-10T06:10:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T06:10:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:10:20.120 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-10T06:10:20.120 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:10:20.120 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:10:20.120 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-02-10 06:10:06.410286983 +0000 2024-02-10T06:10:20.120 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-02-10 06:03:48.589804247 +0000 2024-02-10T06:10:20.120 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-02-10 06:03:48.589804247 +0000 2024-02-10T06:10:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-02-10 06:03:48.589804247 +0000 2024-02-10T06:10:20.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T06:10:20.186 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-02-10T06:10:20.186 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-02-10T06:10:20.186 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000172292 s, 3.0 MB/s 2024-02-10T06:10:20.187 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T06:10:20.244 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-02-10T06:10:20.298 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T06:10:20.299 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:10:20.299 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-10T06:10:20.299 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:10:20.299 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:10:20.299 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-02-10 06:10:06.410286983 +0000 2024-02-10T06:10:20.299 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-02-10 06:03:48.931796285 +0000 2024-02-10T06:10:20.299 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-02-10 06:03:48.931796285 +0000 2024-02-10T06:10:20.299 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-02-10 06:03:48.931796285 +0000 2024-02-10T06:10:20.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T06:10:20.366 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-02-10T06:10:20.366 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-02-10T06:10:20.366 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000191343 s, 2.7 MB/s 2024-02-10T06:10:20.367 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T06:10:20.422 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-02-10T06:10:20.477 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T06:10:20.478 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T06:10:20.478 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-10T06:10:20.478 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:10:20.478 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T06:10:20.478 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-02-10 06:10:06.411286961 +0000 2024-02-10T06:10:20.478 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-02-10 06:03:49.286788020 +0000 2024-02-10T06:10:20.478 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-02-10 06:03:49.286788020 +0000 2024-02-10T06:10:20.478 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-02-10 06:03:49.286788020 +0000 2024-02-10T06:10:20.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T06:10:20.543 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-02-10T06:10:20.543 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-02-10T06:10:20.543 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000163813 s, 3.1 MB/s 2024-02-10T06:10:20.544 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T06:10:20.600 INFO:tasks.cephadm:Deploying osd.0 on smithi089 with /dev/vg_nvme/lv_4... 2024-02-10T06:10:20.601 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T06:10:21.031 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:10:21.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:21 smithi173 ceph-mon[36251]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:21.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:21 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:21.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:21 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:21.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:21 smithi173 ceph-mon[36251]: Reconfiguring crash.smithi089 (monmap changed)... 2024-02-10T06:10:21.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:21 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:10:21.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:21 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:21.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:21 smithi173 ceph-mon[36251]: Reconfiguring daemon crash.smithi089 on smithi089 2024-02-10T06:10:21.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:21 smithi089 ceph-mon[29217]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:21.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:21 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:21.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:21 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:21.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:21 smithi089 ceph-mon[29217]: Reconfiguring crash.smithi089 (monmap changed)... 2024-02-10T06:10:21.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:21 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:10:21.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:21 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:21.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:21 smithi089 ceph-mon[29217]: Reconfiguring daemon crash.smithi089 on smithi089 2024-02-10T06:10:23.341 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:23 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:23.341 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:23 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:23.341 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:23 smithi089 ceph-mon[29217]: Reconfiguring alertmanager.smithi089 (dependencies changed)... 2024-02-10T06:10:23.341 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:23 smithi089 ceph-mon[29217]: Reconfiguring daemon alertmanager.smithi089 on smithi089 2024-02-10T06:10:23.341 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:23 smithi089 ceph-mon[29217]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:23.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:23 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:23.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:23 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:23.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:23 smithi173 ceph-mon[36251]: Reconfiguring alertmanager.smithi089 (dependencies changed)... 2024-02-10T06:10:23.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:23 smithi173 ceph-mon[36251]: Reconfiguring daemon alertmanager.smithi089 on smithi089 2024-02-10T06:10:23.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:23 smithi173 ceph-mon[36251]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:23.921 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T06:10:23.921 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T06:10:23.921 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-02-10T06:10:23.921 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-02-10T06:10:23.921 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116096 s, 903 MB/s 2024-02-10T06:10:23.921 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-02-10T06:10:24.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_4 2024-02-10T06:10:25.331 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:25 smithi089 ceph-mon[29217]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:25.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:25 smithi173 ceph-mon[36251]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:25.772 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.200+0000 7f67cf3f6700 1 -- 172.21.15.89:0/4253636019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67c80fcfb0 msgr2=0x7f67c80fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.200+0000 7f67cf3f6700 1 --2- 172.21.15.89:0/4253636019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67c80fcfb0 0x7f67c80fd420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f67b8004940 tx=0x7f67b8009e30 comp rx=0 tx=0).stop 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.200+0000 7f67cf3f6700 1 -- 172.21.15.89:0/4253636019 shutdown_connections 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.200+0000 7f67cf3f6700 1 --2- 172.21.15.89:0/4253636019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67c80fcfb0 0x7f67c80fd420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.200+0000 7f67cf3f6700 1 --2- 172.21.15.89:0/4253636019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c80fbe40 0x7f67c80fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.200+0000 7f67cf3f6700 1 -- 172.21.15.89:0/4253636019 >> 172.21.15.89:0/4253636019 conn(0x7f67c80f7520 msgr2=0x7f67c80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.200+0000 7f67cf3f6700 1 -- 172.21.15.89:0/4253636019 shutdown_connections 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.200+0000 7f67cf3f6700 1 -- 172.21.15.89:0/4253636019 wait complete. 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cf3f6700 1 Processor -- start 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cf3f6700 1 -- start start 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cf3f6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c80fbe40 0x7f67c80fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:27.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cf3f6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67c80fcfb0 0x7f67c80fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cf3f6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c80fea10 con 0x7f67c80fcfb0 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cf3f6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c80feb80 con 0x7f67c80fbe40 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cc991700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67c80fcfb0 0x7f67c80fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cc991700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67c80fcfb0 0x7f67c80fe380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52654/0 (socket says 172.21.15.89:52654) 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cc991700 1 -- 172.21.15.89:0/2942219061 learned_addr learned my addr 172.21.15.89:0/2942219061 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cc991700 1 -- 172.21.15.89:0/2942219061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c80fbe40 msgr2=0x7f67c80fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cd192700 1 --2- 172.21.15.89:0/2942219061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c80fbe40 0x7f67c80fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cc991700 1 --2- 172.21.15.89:0/2942219061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c80fbe40 0x7f67c80fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cc991700 1 -- 172.21.15.89:0/2942219061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b8005040 con 0x7f67c80fcfb0 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.201+0000 7f67cc991700 1 --2- 172.21.15.89:0/2942219061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67c80fcfb0 0x7f67c80fe380 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f67b8009fd0 tx=0x7f67b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.202+0000 7f67bdffb700 1 -- 172.21.15.89:0/2942219061 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67b800e040 con 0x7f67c80fcfb0 2024-02-10T06:10:27.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.202+0000 7f67bdffb700 1 -- 172.21.15.89:0/2942219061 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67b800bcc0 con 0x7f67c80fcfb0 2024-02-10T06:10:27.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.202+0000 7f67cf3f6700 1 -- 172.21.15.89:0/2942219061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c80fee00 con 0x7f67c80fcfb0 2024-02-10T06:10:27.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.202+0000 7f67bdffb700 1 -- 172.21.15.89:0/2942219061 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67b801c630 con 0x7f67c80fcfb0 2024-02-10T06:10:27.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.202+0000 7f67cf3f6700 1 -- 172.21.15.89:0/2942219061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c8115950 con 0x7f67c80fcfb0 2024-02-10T06:10:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.203+0000 7f67cf3f6700 1 -- 172.21.15.89:0/2942219061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c804bed0 con 0x7f67c80fcfb0 2024-02-10T06:10:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.203+0000 7f67bdffb700 1 -- 172.21.15.89:0/2942219061 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f67b8013980 con 0x7f67c80fcfb0 2024-02-10T06:10:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.204+0000 7f67bdffb700 1 --2- 172.21.15.89:0/2942219061 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f67b4069210 0x7f67b406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:10:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.204+0000 7f67bdffb700 1 -- 172.21.15.89:0/2942219061 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f67b8087540 con 0x7f67c80fcfb0 2024-02-10T06:10:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.204+0000 7f67cd192700 1 --2- 172.21.15.89:0/2942219061 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f67b4069210 0x7f67b406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:10:27.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.205+0000 7f67cd192700 1 --2- 172.21.15.89:0/2942219061 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f67b4069210 0x7f67b406b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f67c4006fd0 tx=0x7f67c4008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:10:27.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.209+0000 7f67bdffb700 1 -- 172.21.15.89:0/2942219061 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f67b80583e0 con 0x7f67c80fcfb0 2024-02-10T06:10:27.327 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:27 smithi089 ceph-mon[29217]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:27.327 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:27 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:27.417+0000 7f67cf3f6700 1 -- 172.21.15.89:0/2942219061 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f67c8066660 con 0x7f67b4069210 2024-02-10T06:10:27.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:27 smithi173 ceph-mon[36251]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:27.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:27 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:28.440 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:28 smithi089 ceph-mon[29217]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:10:28.440 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:28 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:10:28.440 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:28 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:10:28.440 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:28 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:28.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:28 smithi173 ceph-mon[36251]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:10:28.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:28 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:10:28.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:28 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:10:28.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:28 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:29.235 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:29 smithi089 ceph-mon[29217]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:29.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:29 smithi173 ceph-mon[36251]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:31.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:30 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:31.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:30 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:31.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:30 smithi089 ceph-mon[29217]: Reconfiguring grafana.smithi089 (dependencies changed)... 2024-02-10T06:10:31.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:30 smithi089 ceph-mon[29217]: Reconfiguring daemon grafana.smithi089 on smithi089 2024-02-10T06:10:31.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:30 smithi089 ceph-mon[29217]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:31.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:30 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:31.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:30 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:31.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:30 smithi173 ceph-mon[36251]: Reconfiguring grafana.smithi089 (dependencies changed)... 2024-02-10T06:10:31.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:30 smithi173 ceph-mon[36251]: Reconfiguring daemon grafana.smithi089 on smithi089 2024-02-10T06:10:31.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:30 smithi173 ceph-mon[36251]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:32.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:31 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3480126773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fae919bb-103f-402d-8509-1e29a0ebc746"}]: dispatch 2024-02-10T06:10:32.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:31 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3480126773' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fae919bb-103f-402d-8509-1e29a0ebc746"}]': finished 2024-02-10T06:10:32.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:31 smithi089 ceph-mon[29217]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T06:10:32.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:31 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:10:32.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:31 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1697675345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:10:32.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:31 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/3480126773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fae919bb-103f-402d-8509-1e29a0ebc746"}]: dispatch 2024-02-10T06:10:32.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:31 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/3480126773' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fae919bb-103f-402d-8509-1e29a0ebc746"}]': finished 2024-02-10T06:10:32.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:31 smithi173 ceph-mon[36251]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T06:10:32.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:31 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:10:32.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:31 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/1697675345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:10:33.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:32 smithi089 ceph-mon[29217]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:33.149 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:32 smithi173 ceph-mon[36251]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:35.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:35 smithi089 ceph-mon[29217]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:35.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:35 smithi173 ceph-mon[36251]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:37.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:37 smithi089 ceph-mon[29217]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:37.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:37 smithi173 ceph-mon[36251]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:39.223 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:38 smithi089 ceph-mon[29217]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:39.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:38 smithi173 ceph-mon[36251]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:41.216 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:41 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:41.216 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:41 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:41.216 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:41 smithi089 ceph-mon[29217]: Reconfiguring prometheus.smithi089 (dependencies changed)... 2024-02-10T06:10:41.216 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:41 smithi089 ceph-mon[29217]: Reconfiguring daemon prometheus.smithi089 on smithi089 2024-02-10T06:10:41.216 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:41 smithi089 ceph-mon[29217]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:41.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:41 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:41.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:41 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:41.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:41 smithi173 ceph-mon[36251]: Reconfiguring prometheus.smithi089 (dependencies changed)... 2024-02-10T06:10:41.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:41 smithi173 ceph-mon[36251]: Reconfiguring daemon prometheus.smithi089 on smithi089 2024-02-10T06:10:41.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:41 smithi173 ceph-mon[36251]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:42.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:10:42.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:42.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:42 smithi089 ceph-mon[29217]: Deploying daemon osd.0 on smithi089 2024-02-10T06:10:42.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T06:10:42.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:42.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:42 smithi173 ceph-mon[36251]: Deploying daemon osd.0 on smithi089 2024-02-10T06:10:43.231 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:43 smithi089 ceph-mon[29217]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:43.231 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:43 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:43.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:43 smithi173 ceph-mon[36251]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:43.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:43 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:45.273 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:45 smithi089 ceph-mon[29217]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:45.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:45 smithi173 ceph-mon[36251]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:46.751 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:46 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:46.752 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:46 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:46.752 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:46 smithi089 ceph-mon[29217]: Reconfiguring crash.smithi173 (monmap changed)... 2024-02-10T06:10:46.752 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:46 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi173", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:10:46.752 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:46 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:46.752 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:46 smithi089 ceph-mon[29217]: Reconfiguring daemon crash.smithi173 on smithi173 2024-02-10T06:10:46.817 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:46 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:46.817 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:46 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:46.817 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:46 smithi173 ceph-mon[36251]: Reconfiguring crash.smithi173 (monmap changed)... 2024-02-10T06:10:46.818 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:46 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi173", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T06:10:46.818 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:46 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:46.818 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:46 smithi173 ceph-mon[36251]: Reconfiguring daemon crash.smithi173 on smithi173 2024-02-10T06:10:47.520 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:47 smithi173 ceph-mon[36251]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:47.521 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:47 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:47.521 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:47 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:47.521 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:47 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.swdirm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:10:47.521 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:47 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:10:47.521 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:47 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:47.556 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:47 smithi089 ceph-mon[29217]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:47.556 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:47 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:47.556 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:47 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:47.556 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:47 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.swdirm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:10:47.557 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:47 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:10:47.557 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:47 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:48.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:48 smithi173 ceph-mon[36251]: Reconfiguring mgr.smithi173.swdirm (monmap changed)... 2024-02-10T06:10:48.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:48 smithi173 ceph-mon[36251]: Reconfiguring daemon mgr.smithi173.swdirm on smithi173 2024-02-10T06:10:48.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:48 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:48.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:48 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:48.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:48 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:10:48.665 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:48 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:10:48.666 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:48 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:48.769 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:48 smithi089 ceph-mon[29217]: Reconfiguring mgr.smithi173.swdirm (monmap changed)... 2024-02-10T06:10:48.769 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:48 smithi089 ceph-mon[29217]: Reconfiguring daemon mgr.smithi173.swdirm on smithi173 2024-02-10T06:10:48.769 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:48 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:48.769 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:48 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:48.769 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:48 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T06:10:48.769 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:48 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T06:10:48.769 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:48 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:49.764 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:49 smithi173 ceph-mon[36251]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:49.764 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:49 smithi173 ceph-mon[36251]: Reconfiguring mon.smithi173 (monmap changed)... 2024-02-10T06:10:49.764 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:49 smithi173 ceph-mon[36251]: Reconfiguring daemon mon.smithi173 on smithi173 2024-02-10T06:10:49.786 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:49 smithi089 ceph-mon[29217]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:49.786 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:49 smithi089 ceph-mon[29217]: Reconfiguring mon.smithi173 (monmap changed)... 2024-02-10T06:10:49.786 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:49 smithi089 ceph-mon[29217]: Reconfiguring daemon mon.smithi173 on smithi173 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi089.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi089.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi089.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi089.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:10:50.790 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.791 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:10:50.791 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:10:50.791 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi089.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:10:50.791 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi089.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:10:50.791 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.791 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:50.791 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.791 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.896 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.896 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.896 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi089.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi089.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi089.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi089.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi089.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi089.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.897 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:50.898 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:50.898 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:51.692 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:51 smithi089 ceph-mon[29217]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:51.692 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:51 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:51.692 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:51 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:51.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:51 smithi173 ceph-mon[36251]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:51.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:51 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:51.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:51 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:53.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:53 smithi089 ceph-mon[29217]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:53.899 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:53 smithi173 ceph-mon[36251]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:54.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:54 smithi089 ceph-mon[29217]: from='osd.0 [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:10:54.899 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:54 smithi173 ceph-mon[36251]: from='osd.0 [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T06:10:55.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.188+0000 7f67bdffb700 1 -- 172.21.15.89:0/2942219061 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f67c8066660 con 0x7f67b4069210 2024-02-10T06:10:55.189 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 0 on host 'smithi089' 2024-02-10T06:10:55.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.191+0000 7f67cf3f6700 1 -- 172.21.15.89:0/2942219061 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f67b4069210 msgr2=0x7f67b406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:55.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.191+0000 7f67cf3f6700 1 --2- 172.21.15.89:0/2942219061 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f67b4069210 0x7f67b406b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f67c4006fd0 tx=0x7f67c4008040 comp rx=0 tx=0).stop 2024-02-10T06:10:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.191+0000 7f67cf3f6700 1 -- 172.21.15.89:0/2942219061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67c80fcfb0 msgr2=0x7f67c80fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:10:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.191+0000 7f67cf3f6700 1 --2- 172.21.15.89:0/2942219061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67c80fcfb0 0x7f67c80fe380 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f67b8009fd0 tx=0x7f67b8009e30 comp rx=0 tx=0).stop 2024-02-10T06:10:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.191+0000 7f67cf3f6700 1 -- 172.21.15.89:0/2942219061 shutdown_connections 2024-02-10T06:10:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.191+0000 7f67cf3f6700 1 --2- 172.21.15.89:0/2942219061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67c80fbe40 0x7f67c80fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.191+0000 7f67cf3f6700 1 --2- 172.21.15.89:0/2942219061 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f67b4069210 0x7f67b406b6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.191+0000 7f67cf3f6700 1 --2- 172.21.15.89:0/2942219061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67c80fcfb0 0x7f67c80fe380 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:10:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.191+0000 7f67cf3f6700 1 -- 172.21.15.89:0/2942219061 >> 172.21.15.89:0/2942219061 conn(0x7f67c80f7520 msgr2=0x7f67c8065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:10:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.191+0000 7f67cf3f6700 1 -- 172.21.15.89:0/2942219061 shutdown_connections 2024-02-10T06:10:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:10:55.191+0000 7f67cf3f6700 1 -- 172.21.15.89:0/2942219061 wait complete. 2024-02-10T06:10:55.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:55 smithi089 ceph-mon[29217]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:55.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:55 smithi089 ceph-mon[29217]: from='osd.0 [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T06:10:55.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:55 smithi089 ceph-mon[29217]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T06:10:55.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:55 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:10:55.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:55 smithi089 ceph-mon[29217]: from='osd.0 [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-10T06:10:55.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:55 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:55.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:55 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:55.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:55 smithi173 ceph-mon[36251]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:55.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:55 smithi173 ceph-mon[36251]: from='osd.0 [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T06:10:55.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:55 smithi173 ceph-mon[36251]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T06:10:55.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:55 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:10:55.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:55 smithi173 ceph-mon[36251]: from='osd.0 [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-10T06:10:55.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:55 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:55.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:55 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:56.363 DEBUG:teuthology.orchestra.run.smithi089:osd.0> sudo journalctl -f -n 0 -u ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T06:10:56.365 INFO:tasks.cephadm:Deploying osd.1 on smithi089 with /dev/vg_nvme/lv_3... 2024-02-10T06:10:56.366 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T06:10:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:10:56.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:56 smithi089 ceph-mon[29217]: from='osd.0 [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-10T06:10:56.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:56 smithi089 ceph-mon[29217]: osdmap e8: 1 total, 0 up, 1 in 2024-02-10T06:10:56.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:56 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:10:56.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:56 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:10:56.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:56 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:56.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:56 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:56.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:56 smithi173 ceph-mon[36251]: from='osd.0 [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-10T06:10:56.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:56 smithi173 ceph-mon[36251]: osdmap e8: 1 total, 0 up, 1 in 2024-02-10T06:10:56.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:56 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:10:56.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:56 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:10:56.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:56 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:56.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:56 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:10:57.778 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:57 smithi089 ceph-mon[29217]: purged_snaps scrub starts 2024-02-10T06:10:57.778 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:57 smithi089 ceph-mon[29217]: purged_snaps scrub ok 2024-02-10T06:10:57.778 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:57 smithi089 ceph-mon[29217]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:57.778 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:57 smithi089 ceph-mon[29217]: osd.0 [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040] boot 2024-02-10T06:10:57.778 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:57 smithi089 ceph-mon[29217]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T06:10:57.778 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:57 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:10:57.778 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:57 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:57 smithi173 ceph-mon[36251]: purged_snaps scrub starts 2024-02-10T06:10:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:57 smithi173 ceph-mon[36251]: purged_snaps scrub ok 2024-02-10T06:10:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:57 smithi173 ceph-mon[36251]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T06:10:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:57 smithi173 ceph-mon[36251]: osd.0 [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040] boot 2024-02-10T06:10:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:57 smithi173 ceph-mon[36251]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T06:10:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:57 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T06:10:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:57 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:59.069 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:58 smithi089 ceph-mon[29217]: osdmap e10: 1 total, 1 up, 1 in 2024-02-10T06:10:59.070 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:10:58 smithi089 ceph-mon[29217]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:10:59.325 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T06:10:59.325 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T06:10:59.325 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-02-10T06:10:59.326 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-02-10T06:10:59.326 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114734 s, 914 MB/s 2024-02-10T06:10:59.326 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-02-10T06:10:59.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:58 smithi173 ceph-mon[36251]: osdmap e10: 1 total, 1 up, 1 in 2024-02-10T06:10:59.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:10:58 smithi173 ceph-mon[36251]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:00.003 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_3 2024-02-10T06:11:00.689 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:11:01.737 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:01 smithi089 ceph-mon[29217]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:01.737 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:01 smithi089 ceph-mon[29217]: Detected new or changed devices on smithi089 2024-02-10T06:11:01.737 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:01.737 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:01.737 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:01.737 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:01 smithi089 ceph-mon[29217]: Adjusting osd_memory_target on smithi089 to 12864M 2024-02-10T06:11:01.738 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:01.738 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:01.738 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:01.738 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:01.738 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:01.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:01 smithi173 ceph-mon[36251]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:01.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:01 smithi173 ceph-mon[36251]: Detected new or changed devices on smithi089 2024-02-10T06:11:01.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:01.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:01.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:01.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:01 smithi173 ceph-mon[36251]: Adjusting osd_memory_target on smithi089 to 12864M 2024-02-10T06:11:01.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:01.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:01.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:01.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:01.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:02.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.264+0000 7ff5f5509700 1 -- 172.21.15.89:0/2027106927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f01003a0 msgr2=0x7ff5f0102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:11:02.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.264+0000 7ff5f5509700 1 --2- 172.21.15.89:0/2027106927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f01003a0 0x7ff5f0102780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8004940 tx=0x7ff5d8009e30 comp rx=0 tx=0).stop 2024-02-10T06:11:02.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.264+0000 7ff5f5509700 1 -- 172.21.15.89:0/2027106927 shutdown_connections 2024-02-10T06:11:02.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.264+0000 7ff5f5509700 1 --2- 172.21.15.89:0/2027106927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f01003a0 0x7ff5f0102780 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:02.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.264+0000 7ff5f5509700 1 --2- 172.21.15.89:0/2027106927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5f00fda80 0x7ff5f00ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:02.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.264+0000 7ff5f5509700 1 -- 172.21.15.89:0/2027106927 >> 172.21.15.89:0/2027106927 conn(0x7ff5f00f7520 msgr2=0x7ff5f00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:11:02.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.264+0000 7ff5f5509700 1 -- 172.21.15.89:0/2027106927 shutdown_connections 2024-02-10T06:11:02.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.264+0000 7ff5f5509700 1 -- 172.21.15.89:0/2027106927 wait complete. 2024-02-10T06:11:02.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.264+0000 7ff5f5509700 1 Processor -- start 2024-02-10T06:11:02.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.264+0000 7ff5f5509700 1 -- start start 2024-02-10T06:11:02.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5f5509700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5f00fda80 0x7ff5f00fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:11:02.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5f5509700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f01003a0 0x7ff5f00fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:11:02.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5f5509700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f00fbf90 con 0x7ff5f01003a0 2024-02-10T06:11:02.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5f5509700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f00fc100 con 0x7ff5f00fda80 2024-02-10T06:11:02.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5e67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f01003a0 0x7ff5f00fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:11:02.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5e67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f01003a0 0x7ff5f00fb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36380/0 (socket says 172.21.15.89:36380) 2024-02-10T06:11:02.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5e67fc700 1 -- 172.21.15.89:0/2455788173 learned_addr learned my addr 172.21.15.89:0/2455788173 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:11:02.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5eeffd700 1 --2- 172.21.15.89:0/2455788173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5f00fda80 0x7ff5f00fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5e67fc700 1 -- 172.21.15.89:0/2455788173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5f00fda80 msgr2=0x7ff5f00fd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5e67fc700 1 --2- 172.21.15.89:0/2455788173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5f00fda80 0x7ff5f00fd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5e67fc700 1 -- 172.21.15.89:0/2455788173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d8005040 con 0x7ff5f01003a0 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5e67fc700 1 --2- 172.21.15.89:0/2455788173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f01003a0 0x7ff5f00fb990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80093b0 tx=0x7ff5d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5e7fff700 1 -- 172.21.15.89:0/2455788173 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5d800e040 con 0x7ff5f01003a0 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5e7fff700 1 -- 172.21.15.89:0/2455788173 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5d800bcc0 con 0x7ff5f01003a0 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5f5509700 1 -- 172.21.15.89:0/2455788173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f00fc380 con 0x7ff5f01003a0 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5e7fff700 1 -- 172.21.15.89:0/2455788173 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5d801c630 con 0x7ff5f01003a0 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.265+0000 7ff5f5509700 1 -- 172.21.15.89:0/2455788173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f0115950 con 0x7ff5f01003a0 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.266+0000 7ff5f5509700 1 -- 172.21.15.89:0/2455788173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f004bed0 con 0x7ff5f01003a0 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.267+0000 7ff5e7fff700 1 -- 172.21.15.89:0/2455788173 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7ff5d8013980 con 0x7ff5f01003a0 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.267+0000 7ff5e7fff700 1 --2- 172.21.15.89:0/2455788173 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff5dc069210 0x7ff5dc06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:11:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.267+0000 7ff5e7fff700 1 -- 172.21.15.89:0/2455788173 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7ff5d8087670 con 0x7ff5f01003a0 2024-02-10T06:11:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.267+0000 7ff5eeffd700 1 --2- 172.21.15.89:0/2455788173 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff5dc069210 0x7ff5dc06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:11:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.267+0000 7ff5eeffd700 1 --2- 172.21.15.89:0/2455788173 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff5dc069210 0x7ff5dc06b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00fd2a0 tx=0x7ff5e000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:11:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.271+0000 7ff5e7fff700 1 -- 172.21.15.89:0/2455788173 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff5d80582b0 con 0x7ff5f01003a0 2024-02-10T06:11:02.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:02.468+0000 7ff5f5509700 1 -- 172.21.15.89:0/2455788173 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5f0101000 con 0x7ff5dc069210 2024-02-10T06:11:02.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:02 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:11:02.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:02 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:11:02.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:02 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:02.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:02 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:11:02.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:02 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:11:02.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:02 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:03.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:03 smithi089 ceph-mon[29217]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:03.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:03 smithi089 ceph-mon[29217]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:03.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:03.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:03.899 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:03 smithi173 ceph-mon[36251]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:03.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:03 smithi173 ceph-mon[36251]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:03.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:03 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:03.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:03 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:05.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:05 smithi089 ceph-mon[29217]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:05.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:05 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/876934765' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61122d5c-670d-4d4f-8575-d3759758e348"}]: dispatch 2024-02-10T06:11:05.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:05 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/876934765' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61122d5c-670d-4d4f-8575-d3759758e348"}]': finished 2024-02-10T06:11:05.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:05 smithi089 ceph-mon[29217]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T06:11:05.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:05 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:11:05.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:05 smithi173 ceph-mon[36251]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:05.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:05 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/876934765' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61122d5c-670d-4d4f-8575-d3759758e348"}]: dispatch 2024-02-10T06:11:05.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:05 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/876934765' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61122d5c-670d-4d4f-8575-d3759758e348"}]': finished 2024-02-10T06:11:05.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:05 smithi173 ceph-mon[36251]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T06:11:05.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:05 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:11:06.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:06 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3309250718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:11:06.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:06 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/3309250718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:11:07.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:07 smithi089 ceph-mon[29217]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:07.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:07 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:07.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:07 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:07.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:07 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:07.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:07 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:07.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:07 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:07.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:07 smithi173 ceph-mon[36251]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:07.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:07 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:07.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:07 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:07.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:07 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:07.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:07 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:07.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:07 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:09.347 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:08 smithi089 ceph-mon[29217]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:09.399 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:08 smithi173 ceph-mon[36251]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:11.415 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:11 smithi089 ceph-mon[29217]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:11.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:11 smithi173 ceph-mon[36251]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:12.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:13.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:13 smithi089 ceph-mon[29217]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:13.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:11:13.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:13.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:13 smithi089 ceph-mon[29217]: Deploying daemon osd.1 on smithi089 2024-02-10T06:11:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:13 smithi173 ceph-mon[36251]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T06:11:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:13 smithi173 ceph-mon[36251]: Deploying daemon osd.1 on smithi089 2024-02-10T06:11:15.576 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:15 smithi089 ceph-mon[29217]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:15.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:15 smithi173 ceph-mon[36251]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:17.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:17 smithi089 ceph-mon[29217]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:17.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:17 smithi173 ceph-mon[36251]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:18.369 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:18 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:18.369 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:18 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:18.369 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:18 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:18.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:18 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:18.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:18 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:18.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:18 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:19.502 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:19 smithi089 ceph-mon[29217]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:19.502 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:19.502 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:19.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:19 smithi173 ceph-mon[36251]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:19.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:19.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:20.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.822+0000 7ff5e7fff700 1 -- 172.21.15.89:0/2455788173 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff5f0101000 con 0x7ff5dc069210 2024-02-10T06:11:20.823 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 1 on host 'smithi089' 2024-02-10T06:11:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.824+0000 7ff5f5509700 1 -- 172.21.15.89:0/2455788173 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff5dc069210 msgr2=0x7ff5dc06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:11:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.824+0000 7ff5f5509700 1 --2- 172.21.15.89:0/2455788173 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff5dc069210 0x7ff5dc06b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00fd2a0 tx=0x7ff5e000b040 comp rx=0 tx=0).stop 2024-02-10T06:11:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.824+0000 7ff5f5509700 1 -- 172.21.15.89:0/2455788173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f01003a0 msgr2=0x7ff5f00fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:11:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.824+0000 7ff5f5509700 1 --2- 172.21.15.89:0/2455788173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f01003a0 0x7ff5f00fb990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80093b0 tx=0x7ff5d8009e30 comp rx=0 tx=0).stop 2024-02-10T06:11:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.825+0000 7ff5f5509700 1 -- 172.21.15.89:0/2455788173 shutdown_connections 2024-02-10T06:11:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.825+0000 7ff5f5509700 1 --2- 172.21.15.89:0/2455788173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5f00fda80 0x7ff5f00fd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.825+0000 7ff5f5509700 1 --2- 172.21.15.89:0/2455788173 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff5dc069210 0x7ff5dc06b6c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.825+0000 7ff5f5509700 1 --2- 172.21.15.89:0/2455788173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f01003a0 0x7ff5f00fb990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.825+0000 7ff5f5509700 1 -- 172.21.15.89:0/2455788173 >> 172.21.15.89:0/2455788173 conn(0x7ff5f00f7520 msgr2=0x7ff5f00f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:11:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.825+0000 7ff5f5509700 1 -- 172.21.15.89:0/2455788173 shutdown_connections 2024-02-10T06:11:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:20.825+0000 7ff5f5509700 1 -- 172.21.15.89:0/2455788173 wait complete. 2024-02-10T06:11:21.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:21 smithi089 ceph-mon[29217]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:21.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:21 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:21.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:21 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:21.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:21 smithi089 ceph-mon[29217]: from='osd.1 [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:11:21.650 DEBUG:teuthology.orchestra.run.smithi089:osd.1> sudo journalctl -f -n 0 -u ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T06:11:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:21 smithi173 ceph-mon[36251]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:21 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:21 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:21.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:21 smithi173 ceph-mon[36251]: from='osd.1 [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T06:11:21.652 INFO:tasks.cephadm:Deploying osd.2 on smithi089 with /dev/vg_nvme/lv_2... 2024-02-10T06:11:21.653 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T06:11:22.461 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:11:23.025 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:22 smithi089 ceph-mon[29217]: from='osd.1 [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T06:11:23.025 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:22 smithi089 ceph-mon[29217]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T06:11:23.025 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:22 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:11:23.025 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:22 smithi089 ceph-mon[29217]: from='osd.1 [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-10T06:11:23.025 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:22 smithi089 ceph-mon[29217]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:23.025 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:11:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:11:22.793+0000 7fd7c75f1700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T06:11:23.026 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:11:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:11:22.797+0000 7fd7c23ec700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:11:23.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:22 smithi173 ceph-mon[36251]: from='osd.1 [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T06:11:23.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:22 smithi173 ceph-mon[36251]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T06:11:23.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:22 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:11:23.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:22 smithi173 ceph-mon[36251]: from='osd.1 [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-10T06:11:23.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:22 smithi173 ceph-mon[36251]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:24.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:23 smithi089 ceph-mon[29217]: from='osd.1 [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-10T06:11:24.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:23 smithi089 ceph-mon[29217]: osdmap e13: 2 total, 1 up, 2 in 2024-02-10T06:11:24.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:23 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:11:24.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:23 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:11:24.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:23 smithi173 ceph-mon[36251]: from='osd.1 [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-10T06:11:24.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:23 smithi173 ceph-mon[36251]: osdmap e13: 2 total, 1 up, 2 in 2024-02-10T06:11:24.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:23 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:11:24.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:23 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:11:24.836 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T06:11:24.836 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T06:11:24.836 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-02-10T06:11:24.836 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-02-10T06:11:24.836 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124258 s, 844 MB/s 2024-02-10T06:11:24.836 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-02-10T06:11:25.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:24 smithi089 ceph-mon[29217]: purged_snaps scrub starts 2024-02-10T06:11:25.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:24 smithi089 ceph-mon[29217]: purged_snaps scrub ok 2024-02-10T06:11:25.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:24 smithi089 ceph-mon[29217]: osd.1 [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828] boot 2024-02-10T06:11:25.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:24 smithi089 ceph-mon[29217]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T06:11:25.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:24 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:11:25.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:24 smithi089 ceph-mon[29217]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:25.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:24 smithi173 ceph-mon[36251]: purged_snaps scrub starts 2024-02-10T06:11:25.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:24 smithi173 ceph-mon[36251]: purged_snaps scrub ok 2024-02-10T06:11:25.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:24 smithi173 ceph-mon[36251]: osd.1 [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828] boot 2024-02-10T06:11:25.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:24 smithi173 ceph-mon[36251]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T06:11:25.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:24 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T06:11:25.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:24 smithi173 ceph-mon[36251]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-10T06:11:25.471 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_2 2024-02-10T06:11:25.843 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:11:26.086 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:25 smithi089 ceph-mon[29217]: osdmap e15: 2 total, 2 up, 2 in 2024-02-10T06:11:26.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:25 smithi173 ceph-mon[36251]: osdmap e15: 2 total, 2 up, 2 in 2024-02-10T06:11:27.155 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:26 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:27.156 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:26 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:27.156 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:26 smithi089 ceph-mon[29217]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:27.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:26 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:27.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:26 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:27.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:26 smithi173 ceph-mon[36251]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:27.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.992+0000 7f24aff74700 1 -- 172.21.15.89:0/3209223536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24a80fcef0 msgr2=0x7f24a80fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:11:27.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.992+0000 7f24aff74700 1 --2- 172.21.15.89:0/3209223536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24a80fcef0 0x7f24a80fd320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f24a4004940 tx=0x7f24a4009e30 comp rx=0 tx=0).stop 2024-02-10T06:11:27.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.992+0000 7f24aff74700 1 -- 172.21.15.89:0/3209223536 shutdown_connections 2024-02-10T06:11:27.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.992+0000 7f24aff74700 1 --2- 172.21.15.89:0/3209223536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24a80fe0e0 0x7f24a80fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:27.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.992+0000 7f24aff74700 1 --2- 172.21.15.89:0/3209223536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24a80fcef0 0x7f24a80fd320 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:27.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.992+0000 7f24aff74700 1 -- 172.21.15.89:0/3209223536 >> 172.21.15.89:0/3209223536 conn(0x7f24a80f8620 msgr2=0x7f24a80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:11:27.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.992+0000 7f24aff74700 1 -- 172.21.15.89:0/3209223536 shutdown_connections 2024-02-10T06:11:27.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.992+0000 7f24aff74700 1 -- 172.21.15.89:0/3209223536 wait complete. 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.992+0000 7f24aff74700 1 Processor -- start 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.992+0000 7f24aff74700 1 -- start start 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24aff74700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24a80fcef0 0x7f24a810d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24aff74700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24a80fe0e0 0x7f24a810ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24aff74700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a810e560 con 0x7f24a80fe0e0 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24aff74700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a810eda0 con 0x7f24a80fcef0 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24ad50f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24a80fe0e0 0x7f24a810ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24ad50f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24a80fe0e0 0x7f24a810ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56490/0 (socket says 172.21.15.89:56490) 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24ad50f700 1 -- 172.21.15.89:0/771184135 learned_addr learned my addr 172.21.15.89:0/771184135 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24ad50f700 1 -- 172.21.15.89:0/771184135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24a80fcef0 msgr2=0x7f24a810d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24ad50f700 1 --2- 172.21.15.89:0/771184135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24a80fcef0 0x7f24a810d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24ad50f700 1 -- 172.21.15.89:0/771184135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a4005040 con 0x7f24a80fe0e0 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24ad50f700 1 --2- 172.21.15.89:0/771184135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24a80fe0e0 0x7f24a810ded0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f249800b790 tx=0x7f2498009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f249e7fc700 1 -- 172.21.15.89:0/771184135 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f249800f770 con 0x7f24a80fe0e0 2024-02-10T06:11:27.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f249e7fc700 1 -- 172.21.15.89:0/771184135 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f249800fda0 con 0x7f24a80fe0e0 2024-02-10T06:11:27.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f249e7fc700 1 -- 172.21.15.89:0/771184135 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f249800d310 con 0x7f24a80fe0e0 2024-02-10T06:11:27.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24aff74700 1 -- 172.21.15.89:0/771184135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a810f080 con 0x7f24a80fe0e0 2024-02-10T06:11:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.993+0000 7f24aff74700 1 -- 172.21.15.89:0/771184135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a810f5c0 con 0x7f24a80fe0e0 2024-02-10T06:11:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.994+0000 7f24aff74700 1 -- 172.21.15.89:0/771184135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a804bed0 con 0x7f24a80fe0e0 2024-02-10T06:11:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.995+0000 7f249e7fc700 1 -- 172.21.15.89:0/771184135 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f2498010420 con 0x7f24a80fe0e0 2024-02-10T06:11:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.995+0000 7f249e7fc700 1 --2- 172.21.15.89:0/771184135 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2494069210 0x7f249406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:11:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.995+0000 7f249e7fc700 1 -- 172.21.15.89:0/771184135 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f24980863a0 con 0x7f24a80fe0e0 2024-02-10T06:11:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.995+0000 7f24add10700 1 --2- 172.21.15.89:0/771184135 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2494069210 0x7f249406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:11:27.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.996+0000 7f24add10700 1 --2- 172.21.15.89:0/771184135 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2494069210 0x7f249406b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f24a4004f90 tx=0x7f24a4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:11:28.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:27.999+0000 7f249e7fc700 1 -- 172.21.15.89:0/771184135 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2498056ed0 con 0x7f24a80fe0e0 2024-02-10T06:11:28.198 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:27 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:28.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:28.200+0000 7f24aff74700 1 -- 172.21.15.89:0/771184135 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f24a810fb30 con 0x7f2494069210 2024-02-10T06:11:28.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:27 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:29.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:28 smithi173 ceph-mon[36251]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:29.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:28 smithi173 ceph-mon[36251]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:29.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:28 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:11:29.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:28 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:11:29.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:28 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:29.169 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:28 smithi089 ceph-mon[29217]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:29.169 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:28 smithi089 ceph-mon[29217]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:29.169 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:28 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:11:29.169 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:28 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:11:29.169 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:28 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:31 smithi089 ceph-mon[29217]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:31 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:31 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:31 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:31 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:31 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:31 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:31 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:31 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:31 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:31.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:31 smithi173 ceph-mon[36251]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:31.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:31 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:31.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:31 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:31.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:31 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:31.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:31 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:31.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:31 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:31.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:31 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:31.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:31 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:31.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:31 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:31.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:31 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:32.447 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:32 smithi089 ceph-mon[29217]: Detected new or changed devices on smithi089 2024-02-10T06:11:32.447 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:32 smithi089 ceph-mon[29217]: Adjusting osd_memory_target on smithi089 to 6432M 2024-02-10T06:11:32.447 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:32 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2647415223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ffd67bc-829e-449f-b4c3-94a11da55da6"}]: dispatch 2024-02-10T06:11:32.447 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:32 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2647415223' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ffd67bc-829e-449f-b4c3-94a11da55da6"}]': finished 2024-02-10T06:11:32.447 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:32 smithi089 ceph-mon[29217]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T06:11:32.447 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:32 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:32.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:32 smithi173 ceph-mon[36251]: Detected new or changed devices on smithi089 2024-02-10T06:11:32.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:32 smithi173 ceph-mon[36251]: Adjusting osd_memory_target on smithi089 to 6432M 2024-02-10T06:11:32.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:32 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2647415223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ffd67bc-829e-449f-b4c3-94a11da55da6"}]: dispatch 2024-02-10T06:11:32.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:32 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2647415223' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ffd67bc-829e-449f-b4c3-94a11da55da6"}]': finished 2024-02-10T06:11:32.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:32 smithi173 ceph-mon[36251]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T06:11:32.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:32 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:33.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:33 smithi089 ceph-mon[29217]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:33.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:33 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2531007585' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:11:33.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:33 smithi173 ceph-mon[36251]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:33.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:33 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2531007585' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:11:35.511 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:35 smithi089 ceph-mon[29217]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:35.511 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:35 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:35.511 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:35 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:35.511 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:35 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:35.511 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:35 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:35.511 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:35 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:35.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:35 smithi173 ceph-mon[36251]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:35.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:35 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:35.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:35 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:35.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:35 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:35.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:35 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:35.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:35 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:37.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:37 smithi089 ceph-mon[29217]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:37.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:37 smithi173 ceph-mon[36251]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:39.271 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:38 smithi089 ceph-mon[29217]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:39.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:38 smithi173 ceph-mon[36251]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:40.238 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:39 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:11:40.238 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:39 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:40.238 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:39 smithi089 ceph-mon[29217]: Deploying daemon osd.2 on smithi089 2024-02-10T06:11:40.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:39 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T06:11:40.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:39 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:40.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:39 smithi173 ceph-mon[36251]: Deploying daemon osd.2 on smithi089 2024-02-10T06:11:41.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:40 smithi089 ceph-mon[29217]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:41.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:40 smithi173 ceph-mon[36251]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:42.478 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:42.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:43.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:43 smithi089 ceph-mon[29217]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:43.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:43 smithi173 ceph-mon[36251]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:45.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:45 smithi089 ceph-mon[29217]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:45.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:45 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:45.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:45 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:45.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:45 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:45.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:45 smithi173 ceph-mon[36251]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:45.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:45 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:45.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:45 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:45.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:45 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:47.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:47 smithi089 ceph-mon[29217]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:47.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:47 smithi173 ceph-mon[36251]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:47.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.783+0000 7f249e7fc700 1 -- 172.21.15.89:0/771184135 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f24a810fb30 con 0x7f2494069210 2024-02-10T06:11:47.785 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 2 on host 'smithi089' 2024-02-10T06:11:47.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.785+0000 7f24aff74700 1 -- 172.21.15.89:0/771184135 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2494069210 msgr2=0x7f249406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:11:47.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.785+0000 7f24aff74700 1 --2- 172.21.15.89:0/771184135 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2494069210 0x7f249406b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f24a4004f90 tx=0x7f24a4009e30 comp rx=0 tx=0).stop 2024-02-10T06:11:47.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.785+0000 7f24aff74700 1 -- 172.21.15.89:0/771184135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24a80fe0e0 msgr2=0x7f24a810ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:11:47.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.785+0000 7f24aff74700 1 --2- 172.21.15.89:0/771184135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24a80fe0e0 0x7f24a810ded0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f249800b790 tx=0x7f2498009250 comp rx=0 tx=0).stop 2024-02-10T06:11:47.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.785+0000 7f24aff74700 1 -- 172.21.15.89:0/771184135 shutdown_connections 2024-02-10T06:11:47.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.785+0000 7f24aff74700 1 --2- 172.21.15.89:0/771184135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24a80fcef0 0x7f24a810d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:47.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.785+0000 7f24aff74700 1 --2- 172.21.15.89:0/771184135 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2494069210 0x7f249406b6c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:47.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.785+0000 7f24aff74700 1 --2- 172.21.15.89:0/771184135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24a80fe0e0 0x7f24a810ded0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:47.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.785+0000 7f24aff74700 1 -- 172.21.15.89:0/771184135 >> 172.21.15.89:0/771184135 conn(0x7f24a80f8620 msgr2=0x7f24a81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:11:47.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.785+0000 7f24aff74700 1 -- 172.21.15.89:0/771184135 shutdown_connections 2024-02-10T06:11:47.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:11:47.785+0000 7f24aff74700 1 -- 172.21.15.89:0/771184135 wait complete. 2024-02-10T06:11:48.619 DEBUG:teuthology.orchestra.run.smithi089:osd.2> sudo journalctl -f -n 0 -u ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T06:11:48.621 INFO:tasks.cephadm:Deploying osd.3 on smithi173 with /dev/vg_nvme/lv_4... 2024-02-10T06:11:48.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T06:11:48.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:48 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:48.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:48 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:48.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:48 smithi089 ceph-mon[29217]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:48.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:48 smithi089 ceph-mon[29217]: from='osd.2 [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:11:48.892 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi173/config 2024-02-10T06:11:48.911 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:48 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:48.911 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:48 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:48.911 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:48 smithi173 ceph-mon[36251]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:48.911 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:48 smithi173 ceph-mon[36251]: from='osd.2 [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T06:11:50.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:49 smithi173 ceph-mon[36251]: from='osd.2 [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T06:11:50.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:49 smithi173 ceph-mon[36251]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T06:11:50.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:49 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:50.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:49 smithi173 ceph-mon[36251]: from='osd.2 [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-10T06:11:50.249 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:49 smithi089 ceph-mon[29217]: from='osd.2 [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T06:11:50.249 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:49 smithi089 ceph-mon[29217]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T06:11:50.249 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:49 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:50.249 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:49 smithi089 ceph-mon[29217]: from='osd.2 [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-10T06:11:50.249 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:11:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:11:49.984+0000 7f2c44ead700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T06:11:50.249 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:11:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:11:49.992+0000 7f2c3e4a5700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:11:50.794 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T06:11:50.794 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T06:11:50.795 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-02-10T06:11:50.795 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-02-10T06:11:50.795 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112004 s, 936 MB/s 2024-02-10T06:11:50.795 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-02-10T06:11:51.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:50 smithi173 ceph-mon[36251]: from='osd.2 [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-10T06:11:51.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:50 smithi173 ceph-mon[36251]: osdmap e18: 3 total, 2 up, 3 in 2024-02-10T06:11:51.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:51.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:51.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:50 smithi173 ceph-mon[36251]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:51.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:51.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:50 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:51.310 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:50 smithi089 ceph-mon[29217]: from='osd.2 [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-10T06:11:51.310 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:50 smithi089 ceph-mon[29217]: osdmap e18: 3 total, 2 up, 3 in 2024-02-10T06:11:51.310 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:51.310 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:51.310 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:50 smithi089 ceph-mon[29217]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-10T06:11:51.310 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:51.310 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:50 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:51.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-02-10T06:11:51.700 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi173/config 2024-02-10T06:11:52.087 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:51 smithi173 ceph-mon[36251]: purged_snaps scrub starts 2024-02-10T06:11:52.087 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:51 smithi173 ceph-mon[36251]: purged_snaps scrub ok 2024-02-10T06:11:52.087 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:51 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:52.087 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:51 smithi173 ceph-mon[36251]: osd.2 [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379] boot 2024-02-10T06:11:52.087 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:51 smithi173 ceph-mon[36251]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T06:11:52.087 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:51 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:52.328 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:51 smithi089 ceph-mon[29217]: purged_snaps scrub starts 2024-02-10T06:11:52.328 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:51 smithi089 ceph-mon[29217]: purged_snaps scrub ok 2024-02-10T06:11:52.328 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:51 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:52.328 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:51 smithi089 ceph-mon[29217]: osd.2 [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379] boot 2024-02-10T06:11:52.328 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:51 smithi089 ceph-mon[29217]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T06:11:52.329 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:51 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T06:11:53.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.092+0000 7f1ae664b700 1 -- 172.21.15.173:0/1670216121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ae00fe0e0 msgr2=0x7f1ae00fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.092+0000 7f1ae664b700 1 --2- 172.21.15.173:0/1670216121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ae00fe0e0 0x7f1ae00fe550 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0004970 tx=0x7f1ad0009e30 comp rx=0 tx=0).stop 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.092+0000 7f1ae664b700 1 -- 172.21.15.173:0/1670216121 shutdown_connections 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.092+0000 7f1ae664b700 1 --2- 172.21.15.173:0/1670216121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ae00fe0e0 0x7f1ae00fe550 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.092+0000 7f1ae664b700 1 --2- 172.21.15.173:0/1670216121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ae00fcef0 0x7f1ae00fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.092+0000 7f1ae664b700 1 -- 172.21.15.173:0/1670216121 >> 172.21.15.173:0/1670216121 conn(0x7f1ae00f8620 msgr2=0x7f1ae00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.092+0000 7f1ae664b700 1 -- 172.21.15.173:0/1670216121 shutdown_connections 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.092+0000 7f1ae664b700 1 -- 172.21.15.173:0/1670216121 wait complete. 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.093+0000 7f1ae664b700 1 Processor -- start 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.093+0000 7f1ae664b700 1 -- start start 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.093+0000 7f1ae664b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ae00fcef0 0x7f1ae010d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.093+0000 7f1ae664b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ae00fe0e0 0x7f1ae010de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.093+0000 7f1ae664b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae010e430 con 0x7f1ae00fcef0 2024-02-10T06:11:53.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.093+0000 7f1ae664b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae010ecd0 con 0x7f1ae00fe0e0 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.093+0000 7f1adf7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ae00fe0e0 0x7f1ae010de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.093+0000 7f1adf7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ae00fe0e0 0x7f1ae010de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:50372/0 (socket says 172.21.15.173:50372) 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.093+0000 7f1adf7fe700 1 -- 172.21.15.173:0/1476374355 learned_addr learned my addr 172.21.15.173:0/1476374355 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.093+0000 7f1adffff700 1 --2- 172.21.15.173:0/1476374355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ae00fcef0 0x7f1ae010d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.094+0000 7f1adf7fe700 1 -- 172.21.15.173:0/1476374355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ae00fcef0 msgr2=0x7f1ae010d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.094+0000 7f1adf7fe700 1 --2- 172.21.15.173:0/1476374355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ae00fcef0 0x7f1ae010d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.094+0000 7f1adf7fe700 1 -- 172.21.15.173:0/1476374355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad0005040 con 0x7f1ae00fe0e0 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.094+0000 7f1adf7fe700 1 --2- 172.21.15.173:0/1476374355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ae00fe0e0 0x7f1ae010de30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0005010 tx=0x7f1ad0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.094+0000 7f1adcff9700 1 -- 172.21.15.173:0/1476374355 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ad000e040 con 0x7f1ae00fe0e0 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.094+0000 7f1adcff9700 1 -- 172.21.15.173:0/1476374355 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ad0004290 con 0x7f1ae00fe0e0 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.094+0000 7f1ae664b700 1 -- 172.21.15.173:0/1476374355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae010ef50 con 0x7f1ae00fe0e0 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.094+0000 7f1adcff9700 1 -- 172.21.15.173:0/1476374355 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ad001c3f0 con 0x7f1ae00fe0e0 2024-02-10T06:11:53.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.094+0000 7f1ae664b700 1 -- 172.21.15.173:0/1476374355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae010f490 con 0x7f1ae00fe0e0 2024-02-10T06:11:53.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.095+0000 7f1ae664b700 1 -- 172.21.15.173:0/1476374355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae004bed0 con 0x7f1ae00fe0e0 2024-02-10T06:11:53.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.095+0000 7f1adcff9700 1 -- 172.21.15.173:0/1476374355 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f1ad0013420 con 0x7f1ae00fe0e0 2024-02-10T06:11:53.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.096+0000 7f1adcff9700 1 --2- 172.21.15.173:0/1476374355 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1ac8069210 0x7f1ac806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:11:53.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.096+0000 7f1adcff9700 1 -- 172.21.15.173:0/1476374355 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3217+0+0 (secure 0 0 0) 0x7f1ad0087be0 con 0x7f1ae00fe0e0 2024-02-10T06:11:53.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.096+0000 7f1adffff700 1 --2- 172.21.15.173:0/1476374355 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1ac8069210 0x7f1ac806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:11:53.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.096+0000 7f1adffff700 1 --2- 172.21.15.173:0/1476374355 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1ac8069210 0x7f1ac806b6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009b80 tx=0x7f1ad400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:11:53.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.100+0000 7f1adcff9700 1 -- 172.21.15.173:0/1476374355 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1ad0058340 con 0x7f1ae00fe0e0 2024-02-10T06:11:53.290 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:53.288+0000 7f1ae664b700 1 -- 172.21.15.173:0/1476374355 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ae010f8e0 con 0x7f1ac8069210 2024-02-10T06:11:53.290 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:53 smithi173 ceph-mon[36251]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T06:11:53.290 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:53 smithi173 ceph-mon[36251]: pgmap v73: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:11:53.290 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:53 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T06:11:53.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:53 smithi089 ceph-mon[29217]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T06:11:53.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:53 smithi089 ceph-mon[29217]: pgmap v73: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:11:53.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:53 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T06:11:54.299 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T06:11:54.299 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T06:11:54.299 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='client.24115 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:54.300 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:54 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:54.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T06:11:54.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T06:11:54.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:54.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='client.24115 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:11:54.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:11:54.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:11:54.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:54.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:54.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:54.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:54.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:54.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:11:54.356 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:54.356 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:54.356 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:54.356 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:54.356 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:54 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:55.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:55 smithi173 ceph-mon[36251]: Detected new or changed devices on smithi089 2024-02-10T06:11:55.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:55 smithi173 ceph-mon[36251]: Adjusting osd_memory_target on smithi089 to 4288M 2024-02-10T06:11:55.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:55 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:55.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:55 smithi173 ceph-mon[36251]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T06:11:55.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:55 smithi173 ceph-mon[36251]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:11:55.329 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:55 smithi089 ceph-mon[29217]: Detected new or changed devices on smithi089 2024-02-10T06:11:55.329 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:55 smithi089 ceph-mon[29217]: Adjusting osd_memory_target on smithi089 to 4288M 2024-02-10T06:11:55.329 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:55 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:55.329 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:55 smithi089 ceph-mon[29217]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T06:11:55.329 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:55 smithi089 ceph-mon[29217]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:11:55.330 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66844]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:11:55.330 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66844]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:11:55.330 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66844]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:11:55.330 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66844]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:11:55.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66869]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:11:55.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66869]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:11:55.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66869]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:11:55.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66869]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:11:55.974 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66917]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:11:55.974 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66917]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:11:55.974 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66917]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:11:55.974 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66917]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:11:56.227 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66947]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:11:56.227 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66947]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:11:56.227 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66947]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:11:56.228 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:11:55 smithi089 sudo[66947]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:11:56.228 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:56 smithi089 ceph-mon[29217]: osdmap e23: 3 total, 3 up, 3 in 2024-02-10T06:11:56.228 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:56 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:56.228 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:56 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/1800125517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc150631-29f2-4b5d-846d-bffbe2c09cec"}]: dispatch 2024-02-10T06:11:56.229 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:56 smithi089 ceph-mon[29217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc150631-29f2-4b5d-846d-bffbe2c09cec"}]: dispatch 2024-02-10T06:11:56.229 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:56 smithi089 ceph-mon[29217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc150631-29f2-4b5d-846d-bffbe2c09cec"}]': finished 2024-02-10T06:11:56.229 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:56 smithi089 ceph-mon[29217]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T06:11:56.229 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:56 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:11:56.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:56 smithi173 ceph-mon[36251]: osdmap e23: 3 total, 3 up, 3 in 2024-02-10T06:11:56.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:56 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:56.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:56 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/1800125517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc150631-29f2-4b5d-846d-bffbe2c09cec"}]: dispatch 2024-02-10T06:11:56.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:56 smithi173 ceph-mon[36251]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc150631-29f2-4b5d-846d-bffbe2c09cec"}]: dispatch 2024-02-10T06:11:56.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:56 smithi173 ceph-mon[36251]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc150631-29f2-4b5d-846d-bffbe2c09cec"}]': finished 2024-02-10T06:11:56.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:56 smithi173 ceph-mon[36251]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T06:11:56.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:56 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:11:56.501 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[66951]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:11:56.501 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[66951]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:11:56.501 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[66951]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:11:56.501 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[66951]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:11:56.757 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[67000]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:11:56.758 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[67000]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:11:56.758 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[67000]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:11:56.758 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[67000]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:11:57.021 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[67020]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:11:57.021 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[67020]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:11:57.021 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[67020]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:11:57.021 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:56 smithi089 sudo[67020]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:11:57.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:57 smithi089 sudo[67036]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:11:57.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:57 smithi089 sudo[67036]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:11:57.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:57 smithi089 sudo[67036]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:11:57.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:57 smithi089 sudo[67036]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:11:57.523 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:57 smithi173 sudo[39992]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T06:11:57.523 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:57 smithi173 sudo[39992]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:11:57.523 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:57 smithi173 sudo[39992]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:11:57.523 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:57 smithi173 sudo[39992]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:11:57.538 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:11:57.537+0000 7f1adcff9700 1 -- 172.21.15.173:0/1476374355 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1ad0054c80 con 0x7f1ae00fe0e0 2024-02-10T06:11:57.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:57 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/1080787671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:11:57.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:57 smithi089 ceph-mon[29217]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:11:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:57 smithi173 sudo[40179]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T06:11:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:57 smithi173 sudo[40179]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T06:11:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:57 smithi173 sudo[40179]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T06:11:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:57 smithi173 sudo[40179]: pam_unix(sudo:session): session closed for user root 2024-02-10T06:11:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:57 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/1080787671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:11:57.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:57 smithi173 ceph-mon[36251]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:11:58.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:11:58.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:11:58.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-02-10T06:11:58.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:11:58.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-02-10T06:11:58.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:11:58.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:11:58.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:58.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:11:58.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: mgrmap e19: smithi089.pwmrye(active, since 2m), standbys: smithi173.swdirm 2024-02-10T06:11:58.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:58.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:58.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:58.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:58.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:58.857 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:58.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:11:58.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:11:58.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-02-10T06:11:58.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:11:58.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2024-02-10T06:11:58.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T06:11:58.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-02-10T06:11:58.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:58.901 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T06:11:58.902 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: mgrmap e19: smithi089.pwmrye(active, since 2m), standbys: smithi173.swdirm 2024-02-10T06:11:58.902 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:58.902 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:58.902 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:58.902 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:58.902 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:58.902 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:11:59.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:11:59 smithi089 ceph-mon[29217]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:11:59.857 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:11:59 smithi173 ceph-mon[36251]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:01.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:01 smithi089 ceph-mon[29217]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:01.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:01 smithi173 ceph-mon[36251]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:03.852 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:03 smithi173 ceph-mon[36251]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:03.852 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:03 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:12:03.852 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:03 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:03.852 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:03 smithi173 ceph-mon[36251]: Deploying daemon osd.3 on smithi173 2024-02-10T06:12:03.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:03 smithi089 ceph-mon[29217]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:03.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T06:12:03.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:03.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:03 smithi089 ceph-mon[29217]: Deploying daemon osd.3 on smithi173 2024-02-10T06:12:05.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:05 smithi173 ceph-mon[36251]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:05.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:05 smithi089 ceph-mon[29217]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:07.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:07 smithi089 ceph-mon[29217]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:07.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:07 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:07.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:07 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:07.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:07 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:07.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:07 smithi173 ceph-mon[36251]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:07.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:07 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:07.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:07 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:07.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:07 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:09.212 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:08 smithi173 ceph-mon[36251]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:09.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:08 smithi089 ceph-mon[29217]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:10.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.449+0000 7f1adcff9700 1 -- 172.21.15.173:0/1476374355 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1ae010f8e0 con 0x7f1ac8069210 2024-02-10T06:12:10.450 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 3 on host 'smithi173' 2024-02-10T06:12:10.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.452+0000 7f1ae664b700 1 -- 172.21.15.173:0/1476374355 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1ac8069210 msgr2=0x7f1ac806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:12:10.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.452+0000 7f1ae664b700 1 --2- 172.21.15.173:0/1476374355 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1ac8069210 0x7f1ac806b6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009b80 tx=0x7f1ad400b040 comp rx=0 tx=0).stop 2024-02-10T06:12:10.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.452+0000 7f1ae664b700 1 -- 172.21.15.173:0/1476374355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ae00fe0e0 msgr2=0x7f1ae010de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:12:10.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.452+0000 7f1ae664b700 1 --2- 172.21.15.173:0/1476374355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ae00fe0e0 0x7f1ae010de30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0005010 tx=0x7f1ad0009e30 comp rx=0 tx=0).stop 2024-02-10T06:12:10.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.452+0000 7f1ae664b700 1 -- 172.21.15.173:0/1476374355 shutdown_connections 2024-02-10T06:12:10.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.452+0000 7f1ae664b700 1 --2- 172.21.15.173:0/1476374355 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1ac8069210 0x7f1ac806b6c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:10.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.452+0000 7f1ae664b700 1 --2- 172.21.15.173:0/1476374355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ae00fcef0 0x7f1ae010d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:10.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.452+0000 7f1ae664b700 1 --2- 172.21.15.173:0/1476374355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ae00fe0e0 0x7f1ae010de30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:10.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.452+0000 7f1ae664b700 1 -- 172.21.15.173:0/1476374355 >> 172.21.15.173:0/1476374355 conn(0x7f1ae00f8620 msgr2=0x7f1ae01013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:12:10.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.452+0000 7f1ae664b700 1 -- 172.21.15.173:0/1476374355 shutdown_connections 2024-02-10T06:12:10.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:10.452+0000 7f1ae664b700 1 -- 172.21.15.173:0/1476374355 wait complete. 2024-02-10T06:12:11.649 DEBUG:teuthology.orchestra.run.smithi173:osd.3> sudo journalctl -f -n 0 -u ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T06:12:11.651 INFO:tasks.cephadm:Deploying osd.4 on smithi173 with /dev/vg_nvme/lv_3... 2024-02-10T06:12:11.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T06:12:11.675 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:11 smithi173 ceph-mon[36251]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:11.675 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:11 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:11.675 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:11 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:11.675 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:11 smithi173 ceph-mon[36251]: from='osd.3 [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:12:11.675 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:11 smithi173 ceph-mon[36251]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:12:11.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:11 smithi089 ceph-mon[29217]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:11.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:11 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:11.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:11 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:11.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:11 smithi089 ceph-mon[29217]: from='osd.3 [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:12:11.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:11 smithi089 ceph-mon[29217]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T06:12:11.937 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi173/config 2024-02-10T06:12:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:12 smithi173 ceph-mon[36251]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T06:12:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:12 smithi173 ceph-mon[36251]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T06:12:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:12:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:12 smithi173 ceph-mon[36251]: from='osd.3 [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:12:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:12 smithi173 ceph-mon[36251]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:12:12.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:12.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:12.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:12.651 INFO:journalctl@ceph.osd.3.smithi173.stdout:Feb 10 06:12:12 smithi173 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-3[41814]: 2024-02-10T06:12:12.426+0000 7fe803ae7700 -1 osd.3 0 waiting for initial osdmap 2024-02-10T06:12:12.651 INFO:journalctl@ceph.osd.3.smithi173.stdout:Feb 10 06:12:12 smithi173 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-3[41814]: 2024-02-10T06:12:12.431+0000 7fe7fc8de700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:12:12.802 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:12 smithi089 ceph-mon[29217]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T06:12:12.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:12 smithi089 ceph-mon[29217]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T06:12:12.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:12:12.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:12 smithi089 ceph-mon[29217]: from='osd.3 [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:12:12.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:12 smithi089 ceph-mon[29217]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:12:12.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:12.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:12.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:13.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:13 smithi089 ceph-mon[29217]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:13.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:13 smithi089 ceph-mon[29217]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-02-10T06:12:13.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:13 smithi089 ceph-mon[29217]: osdmap e26: 4 total, 3 up, 4 in 2024-02-10T06:12:13.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:12:13.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:12:13.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:13 smithi089 ceph-mon[29217]: osd.3 [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236] boot 2024-02-10T06:12:13.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:13 smithi089 ceph-mon[29217]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T06:12:13.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:13 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:12:13.862 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:13 smithi173 ceph-mon[36251]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-10T06:12:13.862 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:13 smithi173 ceph-mon[36251]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-02-10T06:12:13.862 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:13 smithi173 ceph-mon[36251]: osdmap e26: 4 total, 3 up, 4 in 2024-02-10T06:12:13.862 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:12:13.862 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:12:13.862 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:13 smithi173 ceph-mon[36251]: osd.3 [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236] boot 2024-02-10T06:12:13.862 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:13 smithi173 ceph-mon[36251]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T06:12:13.862 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:13 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T06:12:14.952 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T06:12:14.952 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T06:12:14.952 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-02-10T06:12:14.952 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-02-10T06:12:14.952 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132441 s, 792 MB/s 2024-02-10T06:12:14.953 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-02-10T06:12:15.085 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:14 smithi173 ceph-mon[36251]: purged_snaps scrub starts 2024-02-10T06:12:15.085 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:14 smithi173 ceph-mon[36251]: purged_snaps scrub ok 2024-02-10T06:12:15.085 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:14 smithi173 ceph-mon[36251]: osdmap e28: 4 total, 4 up, 4 in 2024-02-10T06:12:15.085 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:14 smithi173 ceph-mon[36251]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:15.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:14 smithi089 ceph-mon[29217]: purged_snaps scrub starts 2024-02-10T06:12:15.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:14 smithi089 ceph-mon[29217]: purged_snaps scrub ok 2024-02-10T06:12:15.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:14 smithi089 ceph-mon[29217]: osdmap e28: 4 total, 4 up, 4 in 2024-02-10T06:12:15.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:14 smithi089 ceph-mon[29217]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:16.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:15 smithi173 ceph-mon[36251]: osdmap e29: 4 total, 4 up, 4 in 2024-02-10T06:12:16.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:15 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:16.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:15 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:16.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:15 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:12:16.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:15 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:16.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:15 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:16.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:15 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:16.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:15 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:16.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:15 smithi089 ceph-mon[29217]: osdmap e29: 4 total, 4 up, 4 in 2024-02-10T06:12:16.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:15 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:16.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:15 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:16.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:15 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:12:16.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:15 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:16.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:15 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:16.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:15 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:16.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:15 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:16.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-02-10T06:12:16.643 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi173/config 2024-02-10T06:12:17.054 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:16 smithi173 ceph-mon[36251]: Detected new or changed devices on smithi173 2024-02-10T06:12:17.054 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:16 smithi173 ceph-mon[36251]: Adjusting osd_memory_target on smithi173 to 15936M 2024-02-10T06:12:17.054 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:16 smithi173 ceph-mon[36251]: osdmap e30: 4 total, 4 up, 4 in 2024-02-10T06:12:17.055 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:17.055 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:17.055 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:17.055 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:16 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:17.055 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:16 smithi173 ceph-mon[36251]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:17.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:16 smithi089 ceph-mon[29217]: Detected new or changed devices on smithi173 2024-02-10T06:12:17.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:16 smithi089 ceph-mon[29217]: Adjusting osd_memory_target on smithi173 to 15936M 2024-02-10T06:12:17.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:16 smithi089 ceph-mon[29217]: osdmap e30: 4 total, 4 up, 4 in 2024-02-10T06:12:17.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:17.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:17.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:17.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:16 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:17.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:16 smithi089 ceph-mon[29217]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:18.086 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:17 smithi173 ceph-mon[36251]: osdmap e31: 4 total, 4 up, 4 in 2024-02-10T06:12:18.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.104+0000 7f03a13ef700 1 -- 172.21.15.173:0/3408666430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f039c066c90 msgr2=0x7f039c0ffad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:12:18.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.104+0000 7f03a13ef700 1 --2- 172.21.15.173:0/3408666430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f039c066c90 0x7f039c0ffad0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f038c004940 tx=0x7f038c009e30 comp rx=0 tx=0).stop 2024-02-10T06:12:18.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.104+0000 7f03a13ef700 1 -- 172.21.15.173:0/3408666430 shutdown_connections 2024-02-10T06:12:18.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.104+0000 7f03a13ef700 1 --2- 172.21.15.173:0/3408666430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f039c1000a0 0x7f039c102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:18.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.104+0000 7f03a13ef700 1 --2- 172.21.15.173:0/3408666430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f039c066c90 0x7f039c0ffad0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:18.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.104+0000 7f03a13ef700 1 -- 172.21.15.173:0/3408666430 >> 172.21.15.173:0/3408666430 conn(0x7f039c0f7520 msgr2=0x7f039c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:12:18.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.104+0000 7f03a13ef700 1 -- 172.21.15.173:0/3408666430 shutdown_connections 2024-02-10T06:12:18.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.104+0000 7f03a13ef700 1 -- 172.21.15.173:0/3408666430 wait complete. 2024-02-10T06:12:18.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f03a13ef700 1 Processor -- start 2024-02-10T06:12:18.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f03a13ef700 1 -- start start 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f03a13ef700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f039c066c90 0x7f039c10d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f03a13ef700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f039c1000a0 0x7f039c10de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f03a13ef700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c10e460 con 0x7f039c066c90 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f03a13ef700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c10ed00 con 0x7f039c1000a0 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f039a7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f039c1000a0 0x7f039c10de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f039a7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f039c1000a0 0x7f039c10de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:60684/0 (socket says 172.21.15.173:60684) 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f039a7fc700 1 -- 172.21.15.173:0/225844029 learned_addr learned my addr 172.21.15.173:0/225844029 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f039affd700 1 --2- 172.21.15.173:0/225844029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f039c066c90 0x7f039c10d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f039a7fc700 1 -- 172.21.15.173:0/225844029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f039c066c90 msgr2=0x7f039c10d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f039a7fc700 1 --2- 172.21.15.173:0/225844029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f039c066c90 0x7f039c10d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f039a7fc700 1 -- 172.21.15.173:0/225844029 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038c005040 con 0x7f039c1000a0 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.105+0000 7f039a7fc700 1 --2- 172.21.15.173:0/225844029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f039c1000a0 0x7f039c10de60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0390002a50 tx=0x7f03900046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:12:18.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.106+0000 7f03837fe700 1 -- 172.21.15.173:0/225844029 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0390007cc0 con 0x7f039c1000a0 2024-02-10T06:12:18.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.106+0000 7f03837fe700 1 -- 172.21.15.173:0/225844029 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f039000f040 con 0x7f039c1000a0 2024-02-10T06:12:18.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.106+0000 7f03837fe700 1 -- 172.21.15.173:0/225844029 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f039000da30 con 0x7f039c1000a0 2024-02-10T06:12:18.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.106+0000 7f03a13ef700 1 -- 172.21.15.173:0/225844029 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039c10efe0 con 0x7f039c1000a0 2024-02-10T06:12:18.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.106+0000 7f03a13ef700 1 -- 172.21.15.173:0/225844029 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c10f520 con 0x7f039c1000a0 2024-02-10T06:12:18.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.107+0000 7f03a13ef700 1 -- 172.21.15.173:0/225844029 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f039c04bed0 con 0x7f039c1000a0 2024-02-10T06:12:18.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.107+0000 7f03837fe700 1 -- 172.21.15.173:0/225844029 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f039000db90 con 0x7f039c1000a0 2024-02-10T06:12:18.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.108+0000 7f03837fe700 1 --2- 172.21.15.173:0/225844029 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f03840692a0 0x7f038406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:12:18.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.108+0000 7f03837fe700 1 -- 172.21.15.173:0/225844029 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f0390086860 con 0x7f039c1000a0 2024-02-10T06:12:18.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.108+0000 7f039affd700 1 --2- 172.21.15.173:0/225844029 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f03840692a0 0x7f038406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:12:18.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.109+0000 7f039affd700 1 --2- 172.21.15.173:0/225844029 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f03840692a0 0x7f038406b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f038c00b3b0 tx=0x7f038c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:12:18.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.112+0000 7f03837fe700 1 -- 172.21.15.173:0/225844029 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0390056c20 con 0x7f039c1000a0 2024-02-10T06:12:18.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:18.310+0000 7f03a13ef700 1 -- 172.21.15.173:0/225844029 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f039c10f9c0 con 0x7f03840692a0 2024-02-10T06:12:18.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:17 smithi089 ceph-mon[29217]: osdmap e31: 4 total, 4 up, 4 in 2024-02-10T06:12:18.993 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:18 smithi173 ceph-mon[36251]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:18.993 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:18 smithi173 ceph-mon[36251]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:12:18.993 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:18 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:12:18.993 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:18 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:12:18.993 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:18 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:19.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:18 smithi089 ceph-mon[29217]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:19.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:18 smithi089 ceph-mon[29217]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:12:19.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:18 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:12:19.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:18 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:12:19.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:18 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:21 smithi089 ceph-mon[29217]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T06:12:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:21 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/517290166' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccd0c3a-15fc-43e4-852d-f392f991ef61"}]: dispatch 2024-02-10T06:12:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:21 smithi089 ceph-mon[29217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccd0c3a-15fc-43e4-852d-f392f991ef61"}]: dispatch 2024-02-10T06:12:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:21 smithi089 ceph-mon[29217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cccd0c3a-15fc-43e4-852d-f392f991ef61"}]': finished 2024-02-10T06:12:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:21 smithi089 ceph-mon[29217]: osdmap e32: 5 total, 4 up, 5 in 2024-02-10T06:12:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:21 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:21 smithi173 ceph-mon[36251]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T06:12:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:21 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/517290166' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccd0c3a-15fc-43e4-852d-f392f991ef61"}]: dispatch 2024-02-10T06:12:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:21 smithi173 ceph-mon[36251]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccd0c3a-15fc-43e4-852d-f392f991ef61"}]: dispatch 2024-02-10T06:12:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:21 smithi173 ceph-mon[36251]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cccd0c3a-15fc-43e4-852d-f392f991ef61"}]': finished 2024-02-10T06:12:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:21 smithi173 ceph-mon[36251]: osdmap e32: 5 total, 4 up, 5 in 2024-02-10T06:12:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:21 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:22.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:22 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/2704830180' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:12:22.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:22 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/2704830180' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:12:23.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:23 smithi089 ceph-mon[29217]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-10T06:12:23.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:23 smithi173 ceph-mon[36251]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-02-10T06:12:25.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:25 smithi173 ceph-mon[36251]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T06:12:25.431 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:25 smithi089 ceph-mon[29217]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T06:12:27.513 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:27 smithi173 ceph-mon[36251]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-02-10T06:12:27.513 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:27 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:27.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:27 smithi089 ceph-mon[29217]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-02-10T06:12:27.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:27 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:28.291 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:28 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:12:28.292 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:28 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:28.292 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:28 smithi173 ceph-mon[36251]: Deploying daemon osd.4 on smithi173 2024-02-10T06:12:28.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:28 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T06:12:28.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:28 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:28.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:28 smithi089 ceph-mon[29217]: Deploying daemon osd.4 on smithi173 2024-02-10T06:12:29.506 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:29 smithi173 ceph-mon[36251]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T06:12:29.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:29 smithi089 ceph-mon[29217]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T06:12:31.366 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:31 smithi173 ceph-mon[36251]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:31.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:31 smithi089 ceph-mon[29217]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:33 smithi089 ceph-mon[29217]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:33 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:33 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:33 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:33.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:33 smithi173 ceph-mon[36251]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:33.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:33 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:33.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:33 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:33.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:33 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:35.779 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:35 smithi173 ceph-mon[36251]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:35.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.821+0000 7f03837fe700 1 -- 172.21.15.173:0/225844029 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f039c10f9c0 con 0x7f03840692a0 2024-02-10T06:12:35.821 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 4 on host 'smithi173' 2024-02-10T06:12:35.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.823+0000 7f03a13ef700 1 -- 172.21.15.173:0/225844029 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f03840692a0 msgr2=0x7f038406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:12:35.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.823+0000 7f03a13ef700 1 --2- 172.21.15.173:0/225844029 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f03840692a0 0x7f038406b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f038c00b3b0 tx=0x7f038c009e30 comp rx=0 tx=0).stop 2024-02-10T06:12:35.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.823+0000 7f03a13ef700 1 -- 172.21.15.173:0/225844029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f039c1000a0 msgr2=0x7f039c10de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:12:35.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.823+0000 7f03a13ef700 1 --2- 172.21.15.173:0/225844029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f039c1000a0 0x7f039c10de60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0390002a50 tx=0x7f03900046b0 comp rx=0 tx=0).stop 2024-02-10T06:12:35.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.823+0000 7f03a13ef700 1 -- 172.21.15.173:0/225844029 shutdown_connections 2024-02-10T06:12:35.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.823+0000 7f03a13ef700 1 --2- 172.21.15.173:0/225844029 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f03840692a0 0x7f038406b750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:35.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.823+0000 7f03a13ef700 1 --2- 172.21.15.173:0/225844029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f039c066c90 0x7f039c10d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:35.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.823+0000 7f03a13ef700 1 --2- 172.21.15.173:0/225844029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f039c1000a0 0x7f039c10de60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:35.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.823+0000 7f03a13ef700 1 -- 172.21.15.173:0/225844029 >> 172.21.15.173:0/225844029 conn(0x7f039c0f7520 msgr2=0x7f039c0f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:12:35.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.823+0000 7f03a13ef700 1 -- 172.21.15.173:0/225844029 shutdown_connections 2024-02-10T06:12:35.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:35.823+0000 7f03a13ef700 1 -- 172.21.15.173:0/225844029 wait complete. 2024-02-10T06:12:35.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:35 smithi089 ceph-mon[29217]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:36.398 DEBUG:teuthology.orchestra.run.smithi173:osd.4> sudo journalctl -f -n 0 -u ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T06:12:36.400 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/vg_nvme/lv_2... 2024-02-10T06:12:36.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T06:12:36.650 INFO:journalctl@ceph.osd.4.smithi173.stdout:Feb 10 06:12:36 smithi173 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-4[48267]: 2024-02-10T06:12:36.484+0000 7fed9aaeb540 -1 osd.4 0 log_to_monitors true 2024-02-10T06:12:36.808 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi173/config 2024-02-10T06:12:37.078 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:36 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:37.078 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:36 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:37.078 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:36 smithi173 ceph-mon[36251]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:37.078 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:36 smithi173 ceph-mon[36251]: from='osd.4 [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:12:37.078 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:36 smithi173 ceph-mon[36251]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:12:37.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:36 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:37.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:36 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:37.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:36 smithi089 ceph-mon[29217]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:37.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:36 smithi089 ceph-mon[29217]: from='osd.4 [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:12:37.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:36 smithi089 ceph-mon[29217]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T06:12:38.021 INFO:journalctl@ceph.osd.4.smithi173.stdout:Feb 10 06:12:37 smithi173 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-4[48267]: 2024-02-10T06:12:37.827+0000 7fed912bd700 -1 osd.4 0 waiting for initial osdmap 2024-02-10T06:12:38.022 INFO:journalctl@ceph.osd.4.smithi173.stdout:Feb 10 06:12:37 smithi173 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-4[48267]: 2024-02-10T06:12:37.835+0000 7fed8c0b8700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:12:38.022 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:37 smithi173 ceph-mon[36251]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T06:12:38.022 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:37 smithi173 ceph-mon[36251]: osdmap e33: 5 total, 4 up, 5 in 2024-02-10T06:12:38.022 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:37 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:38.022 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:37 smithi173 ceph-mon[36251]: from='osd.4 [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:12:38.022 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:37 smithi173 ceph-mon[36251]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:12:38.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:37 smithi089 ceph-mon[29217]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T06:12:38.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:37 smithi089 ceph-mon[29217]: osdmap e33: 5 total, 4 up, 5 in 2024-02-10T06:12:38.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:37 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:38.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:37 smithi089 ceph-mon[29217]: from='osd.4 [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:12:38.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:37 smithi089 ceph-mon[29217]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:12:38.586 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T06:12:38.587 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T06:12:38.587 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-02-10T06:12:38.587 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-02-10T06:12:38.587 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012474 s, 841 MB/s 2024-02-10T06:12:38.587 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-02-10T06:12:39.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:38 smithi089 ceph-mon[29217]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-02-10T06:12:39.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:38 smithi089 ceph-mon[29217]: osdmap e34: 5 total, 4 up, 5 in 2024-02-10T06:12:39.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:38 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:39.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:38 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:39.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:38 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:39.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:38 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:39.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:38 smithi089 ceph-mon[29217]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:39.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:38 smithi173 ceph-mon[36251]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-02-10T06:12:39.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:38 smithi173 ceph-mon[36251]: osdmap e34: 5 total, 4 up, 5 in 2024-02-10T06:12:39.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:38 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:39.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:38 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:39.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:38 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:39.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:38 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:39.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:38 smithi173 ceph-mon[36251]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-10T06:12:39.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-02-10T06:12:39.897 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi173/config 2024-02-10T06:12:40.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:39 smithi089 ceph-mon[29217]: purged_snaps scrub starts 2024-02-10T06:12:40.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:39 smithi089 ceph-mon[29217]: purged_snaps scrub ok 2024-02-10T06:12:40.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:39 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:40.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:39 smithi089 ceph-mon[29217]: osd.4 [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780] boot 2024-02-10T06:12:40.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:39 smithi089 ceph-mon[29217]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T06:12:40.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:39 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:40.127 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:39 smithi173 ceph-mon[36251]: purged_snaps scrub starts 2024-02-10T06:12:40.127 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:39 smithi173 ceph-mon[36251]: purged_snaps scrub ok 2024-02-10T06:12:40.127 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:39 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:40.127 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:39 smithi173 ceph-mon[36251]: osd.4 [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780] boot 2024-02-10T06:12:40.127 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:39 smithi173 ceph-mon[36251]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T06:12:40.127 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:39 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T06:12:41.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:40 smithi089 ceph-mon[29217]: osdmap e36: 5 total, 5 up, 5 in 2024-02-10T06:12:41.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:40 smithi089 ceph-mon[29217]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:41.358 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:40 smithi173 ceph-mon[36251]: osdmap e36: 5 total, 5 up, 5 in 2024-02-10T06:12:41.358 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:40 smithi173 ceph-mon[36251]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:42.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.511+0000 7fd99b61a700 1 -- 172.21.15.173:0/1514380512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9940fcfb0 msgr2=0x7fd9940fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:12:42.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.511+0000 7fd99b61a700 1 --2- 172.21.15.173:0/1514380512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9940fcfb0 0x7fd9940fd420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd984004940 tx=0x7fd984009e30 comp rx=0 tx=0).stop 2024-02-10T06:12:42.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.511+0000 7fd99b61a700 1 -- 172.21.15.173:0/1514380512 shutdown_connections 2024-02-10T06:12:42.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.511+0000 7fd99b61a700 1 --2- 172.21.15.173:0/1514380512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9940fcfb0 0x7fd9940fd420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:42.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.511+0000 7fd99b61a700 1 --2- 172.21.15.173:0/1514380512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9940fbe40 0x7fd9940fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:42.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.511+0000 7fd99b61a700 1 -- 172.21.15.173:0/1514380512 >> 172.21.15.173:0/1514380512 conn(0x7fd9940f7500 msgr2=0x7fd9940f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:12:42.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.511+0000 7fd99b61a700 1 -- 172.21.15.173:0/1514380512 shutdown_connections 2024-02-10T06:12:42.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.511+0000 7fd99b61a700 1 -- 172.21.15.173:0/1514380512 wait complete. 2024-02-10T06:12:42.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd99b61a700 1 Processor -- start 2024-02-10T06:12:42.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd99b61a700 1 -- start start 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd99b61a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9940fbe40 0x7fd994110650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd99b61a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9940fcfb0 0x7fd994112ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd99b61a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd994066da0 con 0x7fd9940fbe40 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd99b61a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd994066ee0 con 0x7fd9940fcfb0 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd998bb5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9940fcfb0 0x7fd994112ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd998bb5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9940fcfb0 0x7fd994112ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:38780/0 (socket says 172.21.15.173:38780) 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd998bb5700 1 -- 172.21.15.173:0/872741936 learned_addr learned my addr 172.21.15.173:0/872741936 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd9993b6700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9940fbe40 0x7fd994110650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd998bb5700 1 -- 172.21.15.173:0/872741936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9940fbe40 msgr2=0x7fd994110650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd998bb5700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9940fbe40 0x7fd994110650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd998bb5700 1 -- 172.21.15.173:0/872741936 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd984005040 con 0x7fd9940fcfb0 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd9993b6700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9940fbe40 0x7fd994110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:12:42.516 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.512+0000 7fd998bb5700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9940fcfb0 0x7fd994112ba0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd9840093b0 tx=0x7fd984009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:12:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.513+0000 7fd989ffb700 1 -- 172.21.15.173:0/872741936 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd98400e040 con 0x7fd9940fcfb0 2024-02-10T06:12:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.513+0000 7fd989ffb700 1 -- 172.21.15.173:0/872741936 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd98400bcc0 con 0x7fd9940fcfb0 2024-02-10T06:12:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.513+0000 7fd99b61a700 1 -- 172.21.15.173:0/872741936 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9941131f0 con 0x7fd9940fcfb0 2024-02-10T06:12:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.513+0000 7fd989ffb700 1 -- 172.21.15.173:0/872741936 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd98401c630 con 0x7fd9940fcfb0 2024-02-10T06:12:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.513+0000 7fd99b61a700 1 -- 172.21.15.173:0/872741936 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd994113730 con 0x7fd9940fcfb0 2024-02-10T06:12:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.514+0000 7fd99b61a700 1 -- 172.21.15.173:0/872741936 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99404bed0 con 0x7fd9940fcfb0 2024-02-10T06:12:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.515+0000 7fd989ffb700 1 -- 172.21.15.173:0/872741936 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd9840139f0 con 0x7fd9940fcfb0 2024-02-10T06:12:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.515+0000 7fd989ffb700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd9800692a0 0x7fd98006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:12:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.515+0000 7fd989ffb700 1 -- 172.21.15.173:0/872741936 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7fd984087f90 con 0x7fd9940fcfb0 2024-02-10T06:12:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.515+0000 7fd9993b6700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd9800692a0 0x7fd98006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:12:42.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.516+0000 7fd9993b6700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd9800692a0 0x7fd98006b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd994066990 tx=0x7fd990007510 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:12:42.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.520+0000 7fd989ffb700 1 -- 172.21.15.173:0/872741936 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd984054850 con 0x7fd9940fcfb0 2024-02-10T06:12:42.714 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:12:42.712+0000 7fd99b61a700 1 -- 172.21.15.173:0/872741936 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd994101990 con 0x7fd9800692a0 2024-02-10T06:12:43.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: Detected new or changed devices on smithi173 2024-02-10T06:12:43.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:43.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:43.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:12:43.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:12:43.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: Adjusting osd_memory_target on smithi173 to 7968M 2024-02-10T06:12:43.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:43.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:43.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:43.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:43.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:43.106 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:43.106 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:43.106 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:43.106 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:43.106 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:43.126 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: Detected new or changed devices on smithi173 2024-02-10T06:12:43.126 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:43.126 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:43.126 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:12:43.126 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:12:43.126 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: Adjusting osd_memory_target on smithi173 to 7968M 2024-02-10T06:12:43.126 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:43.126 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:43.126 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:43.126 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:43.126 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:43.127 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:43.127 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:43.127 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:43.127 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:43.127 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:44.020 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:43 smithi173 ceph-mon[36251]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:12:44.020 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:43 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:12:44.020 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:43 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:12:44.020 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:43 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:44.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:43 smithi089 ceph-mon[29217]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:12:44.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:43 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T06:12:44.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:43 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T06:12:44.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:43 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:45.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:44 smithi089 ceph-mon[29217]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:45.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:44 smithi173 ceph-mon[36251]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:46.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:45 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/3441306733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00386df2-a6d8-42e8-adfe-58b6141471f5"}]: dispatch 2024-02-10T06:12:46.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:45 smithi089 ceph-mon[29217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00386df2-a6d8-42e8-adfe-58b6141471f5"}]: dispatch 2024-02-10T06:12:46.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:45 smithi089 ceph-mon[29217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00386df2-a6d8-42e8-adfe-58b6141471f5"}]': finished 2024-02-10T06:12:46.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:45 smithi089 ceph-mon[29217]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T06:12:46.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:45 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:12:46.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:45 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/3689571203' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:12:46.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:45 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/3441306733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00386df2-a6d8-42e8-adfe-58b6141471f5"}]: dispatch 2024-02-10T06:12:46.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:45 smithi173 ceph-mon[36251]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00386df2-a6d8-42e8-adfe-58b6141471f5"}]: dispatch 2024-02-10T06:12:46.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:45 smithi173 ceph-mon[36251]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00386df2-a6d8-42e8-adfe-58b6141471f5"}]': finished 2024-02-10T06:12:46.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:45 smithi173 ceph-mon[36251]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T06:12:46.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:45 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:12:46.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:45 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/3689571203' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T06:12:47.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:46 smithi089 ceph-mon[29217]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:47.149 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:46 smithi173 ceph-mon[36251]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:49.236 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:48 smithi173 ceph-mon[36251]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:49.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:48 smithi089 ceph-mon[29217]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:51.280 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:51 smithi173 ceph-mon[36251]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:51.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:51 smithi089 ceph-mon[29217]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:52.543 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:52 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:12:52.544 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:52 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:52.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:52 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T06:12:52.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:52 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:53.474 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:53 smithi173 ceph-mon[36251]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:53.474 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:53 smithi173 ceph-mon[36251]: Deploying daemon osd.5 on smithi173 2024-02-10T06:12:53.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:53 smithi089 ceph-mon[29217]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:53.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:53 smithi089 ceph-mon[29217]: Deploying daemon osd.5 on smithi173 2024-02-10T06:12:55.541 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:55 smithi173 ceph-mon[36251]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:55.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:55 smithi089 ceph-mon[29217]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:57.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:57 smithi173 ceph-mon[36251]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:57.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:57 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:57.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:57 smithi089 ceph-mon[29217]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:57.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:57 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:58.547 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:58.547 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:58.547 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:58 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:58.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:58.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:12:58.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:58 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:59.503 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:12:59 smithi173 ceph-mon[36251]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:12:59.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:12:59 smithi089 ceph-mon[29217]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:13:00.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.327+0000 7fd989ffb700 1 -- 172.21.15.173:0/872741936 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd994101990 con 0x7fd9800692a0 2024-02-10T06:13:00.328 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2024-02-10T06:13:00.330 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.330+0000 7fd99b61a700 1 -- 172.21.15.173:0/872741936 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd9800692a0 msgr2=0x7fd98006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:00.330 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.330+0000 7fd99b61a700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd9800692a0 0x7fd98006b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd994066990 tx=0x7fd990007510 comp rx=0 tx=0).stop 2024-02-10T06:13:00.330 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.330+0000 7fd99b61a700 1 -- 172.21.15.173:0/872741936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9940fcfb0 msgr2=0x7fd994112ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:00.330 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.330+0000 7fd99b61a700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9940fcfb0 0x7fd994112ba0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd9840093b0 tx=0x7fd984009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:00.330 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.330+0000 7fd99b61a700 1 -- 172.21.15.173:0/872741936 shutdown_connections 2024-02-10T06:13:00.330 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.330+0000 7fd99b61a700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd9800692a0 0x7fd98006b750 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:00.331 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.330+0000 7fd99b61a700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9940fbe40 0x7fd994110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:00.331 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.330+0000 7fd99b61a700 1 --2- 172.21.15.173:0/872741936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9940fcfb0 0x7fd994112ba0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:00.331 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.330+0000 7fd99b61a700 1 -- 172.21.15.173:0/872741936 >> 172.21.15.173:0/872741936 conn(0x7fd9940f7500 msgr2=0x7fd994100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:00.331 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.330+0000 7fd99b61a700 1 -- 172.21.15.173:0/872741936 shutdown_connections 2024-02-10T06:13:00.331 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:00.330+0000 7fd99b61a700 1 -- 172.21.15.173:0/872741936 wait complete. 2024-02-10T06:13:01.412 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T06:13:01.414 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-02-10T06:13:01.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T06:13:01.437 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:01 smithi173 ceph-mon[36251]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:13:01.438 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:01.438 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:01.438 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:01 smithi173 ceph-mon[36251]: from='osd.5 [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:13:01.438 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:01 smithi173 ceph-mon[36251]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:13:01.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:01 smithi089 ceph-mon[29217]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:13:01.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:01.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:01.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:01 smithi089 ceph-mon[29217]: from='osd.5 [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:13:01.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:01 smithi089 ceph-mon[29217]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T06:13:01.685 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:02.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:02 smithi089 ceph-mon[29217]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T06:13:02.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:02 smithi089 ceph-mon[29217]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T06:13:02.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:02 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:13:02.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:02 smithi089 ceph-mon[29217]: from='osd.5 [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:13:02.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:02 smithi089 ceph-mon[29217]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:13:02.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:02 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:02.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:02 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:02.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:02 smithi173 ceph-mon[36251]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T06:13:02.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:02 smithi173 ceph-mon[36251]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T06:13:02.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:02 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:13:02.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:02 smithi173 ceph-mon[36251]: from='osd.5 [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:13:02.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:02 smithi173 ceph-mon[36251]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-02-10T06:13:02.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:02 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:02.642 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:02 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:02.643 INFO:journalctl@ceph.osd.5.smithi173.stdout:Feb 10 06:13:02 smithi173 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-5[55161]: 2024-02-10T06:13:02.316+0000 7fc664216700 -1 osd.5 0 waiting for initial osdmap 2024-02-10T06:13:02.643 INFO:journalctl@ceph.osd.5.smithi173.stdout:Feb 10 06:13:02 smithi173 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-5[55161]: 2024-02-10T06:13:02.323+0000 7fc65d00d700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T06:13:03.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.343+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/2703753055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc0fbe40 msgr2=0x7fb4bc0fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.343+0000 7fb4c2f6d700 1 --2- 172.21.15.89:0/2703753055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc0fbe40 0x7fb4bc0fc250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0004940 tx=0x7fb4b0009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.343+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/2703753055 shutdown_connections 2024-02-10T06:13:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.343+0000 7fb4c2f6d700 1 --2- 172.21.15.89:0/2703753055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4bc0fcfb0 0x7fb4bc0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.343+0000 7fb4c2f6d700 1 --2- 172.21.15.89:0/2703753055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc0fbe40 0x7fb4bc0fc250 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.343+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/2703753055 >> 172.21.15.89:0/2703753055 conn(0x7fb4bc0f74e0 msgr2=0x7fb4bc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.343+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/2703753055 shutdown_connections 2024-02-10T06:13:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.343+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/2703753055 wait complete. 2024-02-10T06:13:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.343+0000 7fb4c2f6d700 1 Processor -- start 2024-02-10T06:13:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.343+0000 7fb4c2f6d700 1 -- start start 2024-02-10T06:13:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4c2f6d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc0fbe40 0x7fb4bc10c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4c2f6d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4bc0fcfb0 0x7fb4bc10cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4c2f6d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc10d380 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4c2f6d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc10dc20 con 0x7fb4bc0fcfb0 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4c0d09700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc0fbe40 0x7fb4bc10c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4c0d09700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc0fbe40 0x7fb4bc10c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34838/0 (socket says 172.21.15.89:34838) 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4c0d09700 1 -- 172.21.15.89:0/4230205939 learned_addr learned my addr 172.21.15.89:0/4230205939 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4bbfff700 1 --2- 172.21.15.89:0/4230205939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4bc0fcfb0 0x7fb4bc10cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4c0d09700 1 -- 172.21.15.89:0/4230205939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4bc0fcfb0 msgr2=0x7fb4bc10cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4c0d09700 1 --2- 172.21.15.89:0/4230205939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4bc0fcfb0 0x7fb4bc10cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4c0d09700 1 -- 172.21.15.89:0/4230205939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b0005040 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4c0d09700 1 --2- 172.21.15.89:0/4230205939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc0fbe40 0x7fb4bc10c840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00093b0 tx=0x7fb4b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.344+0000 7fb4b97fa700 1 -- 172.21.15.89:0/4230205939 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b000e040 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.345+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/4230205939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4bc10dea0 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.345+0000 7fb4b97fa700 1 -- 172.21.15.89:0/4230205939 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4b000bcc0 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.345+0000 7fb4b97fa700 1 -- 172.21.15.89:0/4230205939 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b001c630 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.345+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/4230205939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4bc10e3e0 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.345+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/4230205939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4bc04bed0 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.346+0000 7fb4b97fa700 1 -- 172.21.15.89:0/4230205939 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb4b0013980 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.346+0000 7fb4b97fa700 1 --2- 172.21.15.89:0/4230205939 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fb4ac06b4b0 0x7fb4ac06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:03.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.346+0000 7fb4b97fa700 1 -- 172.21.15.89:0/4230205939 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb4b00881c0 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.346+0000 7fb4bbfff700 1 --2- 172.21.15.89:0/4230205939 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fb4ac06b4b0 0x7fb4ac06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:03.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.347+0000 7fb4bbfff700 1 --2- 172.21.15.89:0/4230205939 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fb4ac06b4b0 0x7fb4ac06d960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8007fd0 tx=0x7fb4a8009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:03.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.351+0000 7fb4b97fa700 1 -- 172.21.15.89:0/4230205939 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb4b00582c0 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.530 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:03 smithi173 ceph-mon[36251]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:13:03.530 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:03 smithi173 ceph-mon[36251]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-02-10T06:13:03.530 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:03 smithi173 ceph-mon[36251]: osdmap e39: 6 total, 5 up, 6 in 2024-02-10T06:13:03.531 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:03 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:13:03.531 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:03 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:13:03.531 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:03 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:13:03.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.536+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/4230205939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fb4bc104930 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.537 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:03 smithi089 ceph-mon[29217]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:13:03.537 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:03 smithi089 ceph-mon[29217]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-02-10T06:13:03.537 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:03 smithi089 ceph-mon[29217]: osdmap e39: 6 total, 5 up, 6 in 2024-02-10T06:13:03.537 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:13:03.537 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:13:03.537 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:03 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:13:03.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.536+0000 7fb4b97fa700 1 -- 172.21.15.89:0/4230205939 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7fb4b0057e50 con 0x7fb4bc0fbe40 2024-02-10T06:13:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:13:03.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.538+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/4230205939 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fb4ac06b4b0 msgr2=0x7fb4ac06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:03.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.538+0000 7fb4c2f6d700 1 --2- 172.21.15.89:0/4230205939 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fb4ac06b4b0 0x7fb4ac06d960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8007fd0 tx=0x7fb4a8009040 comp rx=0 tx=0).stop 2024-02-10T06:13:03.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.538+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/4230205939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc0fbe40 msgr2=0x7fb4bc10c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:03.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.539+0000 7fb4c2f6d700 1 --2- 172.21.15.89:0/4230205939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc0fbe40 0x7fb4bc10c840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00093b0 tx=0x7fb4b0009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.539+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/4230205939 shutdown_connections 2024-02-10T06:13:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.539+0000 7fb4c2f6d700 1 --2- 172.21.15.89:0/4230205939 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fb4ac06b4b0 0x7fb4ac06d960 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.539+0000 7fb4c2f6d700 1 --2- 172.21.15.89:0/4230205939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4bc0fbe40 0x7fb4bc10c840 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.539+0000 7fb4c2f6d700 1 --2- 172.21.15.89:0/4230205939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4bc0fcfb0 0x7fb4bc10cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.539+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/4230205939 >> 172.21.15.89:0/4230205939 conn(0x7fb4bc0f74e0 msgr2=0x7fb4bc100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.539+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/4230205939 shutdown_connections 2024-02-10T06:13:03.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:03.539+0000 7fb4c2f6d700 1 -- 172.21.15.89:0/4230205939 wait complete. 2024-02-10T06:13:04.252 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1707545583,"num_in_osds":6,"osd_in_since":1707545565,"num_remapped_pgs":0} 2024-02-10T06:13:04.253 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T06:13:04.533 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:04.541 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:04 smithi089 ceph-mon[29217]: purged_snaps scrub starts 2024-02-10T06:13:04.541 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:04 smithi089 ceph-mon[29217]: purged_snaps scrub ok 2024-02-10T06:13:04.541 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:04 smithi089 ceph-mon[29217]: osd.5 [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782] boot 2024-02-10T06:13:04.541 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:04 smithi089 ceph-mon[29217]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T06:13:04.541 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:04 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:13:04.541 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:04 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/4230205939' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T06:13:04.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:04 smithi173 ceph-mon[36251]: purged_snaps scrub starts 2024-02-10T06:13:04.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:04 smithi173 ceph-mon[36251]: purged_snaps scrub ok 2024-02-10T06:13:04.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:04 smithi173 ceph-mon[36251]: osd.5 [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782] boot 2024-02-10T06:13:04.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:04 smithi173 ceph-mon[36251]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T06:13:04.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:04 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T06:13:04.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:04 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/4230205939' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T06:13:05.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:13:05.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T06:13:05.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:05.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:05.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:13:05.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:13:05.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:13:05.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:05.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:13:05.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:13:05.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:05.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:05 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:13:05.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-10T06:13:05.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T06:13:05.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:05.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:05.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:13:05.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:13:05.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:13:05.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:05.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:13:05.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:13:05.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:05.651 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:05 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:13:06.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.083+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/1785161572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd080702d0 msgr2=0x7fcd080706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.083+0000 7fcd0ecb5700 1 --2- 172.21.15.89:0/1785161572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd080702d0 0x7fcd080706a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009d10 tx=0x7fcd00004d10 comp rx=0 tx=0).stop 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.083+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/1785161572 shutdown_connections 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.083+0000 7fcd0ecb5700 1 --2- 172.21.15.89:0/1785161572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd08070be0 0x7fcd081092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.083+0000 7fcd0ecb5700 1 --2- 172.21.15.89:0/1785161572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd080702d0 0x7fcd080706a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.083+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/1785161572 >> 172.21.15.89:0/1785161572 conn(0x7fcd080f8ae0 msgr2=0x7fcd080faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.083+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/1785161572 shutdown_connections 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.083+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/1785161572 wait complete. 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.083+0000 7fcd0ecb5700 1 Processor -- start 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ecb5700 1 -- start start 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ecb5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd080702d0 0x7fcd08115ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ecb5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd08070be0 0x7fcd08116420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ecb5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd08116a70 con 0x7fcd080702d0 2024-02-10T06:13:06.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ecb5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0810fd80 con 0x7fcd08070be0 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ca51700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd080702d0 0x7fcd08115ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ca51700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd080702d0 0x7fcd08115ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34866/0 (socket says 172.21.15.89:34866) 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ca51700 1 -- 172.21.15.89:0/2582945010 learned_addr learned my addr 172.21.15.89:0/2582945010 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd07fff700 1 --2- 172.21.15.89:0/2582945010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd08070be0 0x7fcd08116420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ca51700 1 -- 172.21.15.89:0/2582945010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd08070be0 msgr2=0x7fcd08116420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ca51700 1 --2- 172.21.15.89:0/2582945010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd08070be0 0x7fcd08116420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ca51700 1 -- 172.21.15.89:0/2582945010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd00009940 con 0x7fcd080702d0 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.084+0000 7fcd0ca51700 1 --2- 172.21.15.89:0/2582945010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd080702d0 0x7fcd08115ee0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcd00008dd0 tx=0x7fcd00004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.085+0000 7fcd057fa700 1 -- 172.21.15.89:0/2582945010 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd0000e040 con 0x7fcd080702d0 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.085+0000 7fcd057fa700 1 -- 172.21.15.89:0/2582945010 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd000128e0 con 0x7fcd080702d0 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.085+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/2582945010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd08110000 con 0x7fcd080702d0 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.085+0000 7fcd057fa700 1 -- 172.21.15.89:0/2582945010 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd0001c880 con 0x7fcd080702d0 2024-02-10T06:13:06.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.085+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/2582945010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd08110540 con 0x7fcd080702d0 2024-02-10T06:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.086+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/2582945010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd08106980 con 0x7fcd080702d0 2024-02-10T06:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.086+0000 7fcd057fa700 1 -- 172.21.15.89:0/2582945010 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcd00012410 con 0x7fcd080702d0 2024-02-10T06:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.086+0000 7fcd057fa700 1 --2- 172.21.15.89:0/2582945010 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fccf40692a0 0x7fccf406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:06.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.086+0000 7fcd057fa700 1 -- 172.21.15.89:0/2582945010 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcd00088c10 con 0x7fcd080702d0 2024-02-10T06:13:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.086+0000 7fcd07fff700 1 --2- 172.21.15.89:0/2582945010 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fccf40692a0 0x7fccf406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:06.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.087+0000 7fcd07fff700 1 --2- 172.21.15.89:0/2582945010 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fccf40692a0 0x7fccf406b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009fd0 tx=0x7fccf8009f20 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:06.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.091+0000 7fcd057fa700 1 -- 172.21.15.89:0/2582945010 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcd00058c90 con 0x7fcd080702d0 2024-02-10T06:13:06.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.275+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/2582945010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd0804bed0 con 0x7fcd080702d0 2024-02-10T06:13:06.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.276+0000 7fcd057fa700 1 -- 172.21.15.89:0/2582945010 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11058 (secure 0 0 0) 0x7fcd00004e10 con 0x7fcd080702d0 2024-02-10T06:13:06.277 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:13:06.277 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":41,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","created":"2024-02-10T06:07:41.156060+0000","modified":"2024-02-10T06:13:04.323413+0000","last_up_change":"2024-02-10T06:13:03.312644+0000","last_in_change":"2024-02-10T06:12:45.108728+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T06:11:52.416365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fae919bb-103f-402d-8509-1e29a0ebc746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6803","nonce":2704721040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6805","nonce":2704721040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6809","nonce":2704721040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6807","nonce":2704721040}]},"public_addr":"172.21.15.89:6803/2704721040","cluster_addr":"172.21.15.89:6805/2704721040","heartbeat_back_addr":"172.21.15.89:6809/2704721040","heartbeat_front_addr":"172.21.15.89:6807/2704721040","state":["exists","up"]},{"osd":1,"uuid":"61122d5c-670d-4d4f-8575-d3759758e348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6811","nonce":3051839828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6813","nonce":3051839828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6817","nonce":3051839828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6815","nonce":3051839828}]},"public_addr":"172.21.15.89:6811/3051839828","cluster_addr":"172.21.15.89:6813/3051839828","heartbeat_back_addr":"172.21.15.89:6817/3051839828","heartbeat_front_addr":"172.21.15.89:6815/3051839828","state":["exists","up"]},{"osd":2,"uuid":"3ffd67bc-829e-449f-b4c3-94a11da55da6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6819","nonce":637372379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6821","nonce":637372379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6825","nonce":637372379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6823","nonce":637372379}]},"public_addr":"172.21.15.89:6819/637372379","cluster_addr":"172.21.15.89:6821/637372379","heartbeat_back_addr":"172.21.15.89:6825/637372379","heartbeat_front_addr":"172.21.15.89:6823/637372379","state":["exists","up"]},{"osd":3,"uuid":"cc150631-29f2-4b5d-846d-bffbe2c09cec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6801","nonce":2644357236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6803","nonce":2644357236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6807","nonce":2644357236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6805","nonce":2644357236}]},"public_addr":"172.21.15.173:6801/2644357236","cluster_addr":"172.21.15.173:6803/2644357236","heartbeat_back_addr":"172.21.15.173:6807/2644357236","heartbeat_front_addr":"172.21.15.173:6805/2644357236","state":["exists","up"]},{"osd":4,"uuid":"cccd0c3a-15fc-43e4-852d-f392f991ef61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6809","nonce":2725914780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6811","nonce":2725914780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6815","nonce":2725914780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6813","nonce":2725914780}]},"public_addr":"172.21.15.173:6809/2725914780","cluster_addr":"172.21.15.173:6811/2725914780","heartbeat_back_addr":"172.21.15.173:6815/2725914780","heartbeat_front_addr":"172.21.15.173:6813/2725914780","state":["exists","up"]},{"osd":5,"uuid":"00386df2-a6d8-42e8-adfe-58b6141471f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6817","nonce":49288782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6819","nonce":49288782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6823","nonce":49288782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6821","nonce":49288782}]},"public_addr":"172.21.15.173:6817/49288782","cluster_addr":"172.21.15.173:6819/49288782","heartbeat_back_addr":"172.21.15.173:6823/49288782","heartbeat_front_addr":"172.21.15.173:6821/49288782","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:10:55.172572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:11:22.140314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:11:49.151682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:12:12.352626+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:12:37.457216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:13:02.197643+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/1955086590":"2024-02-11T06:09:42.089772+0000","172.21.15.89:0/644693661":"2024-02-11T06:09:42.089772+0000","172.21.15.89:0/1015341098":"2024-02-11T06:09:42.089772+0000","172.21.15.89:6800/1955086590":"2024-02-11T06:09:42.089772+0000","172.21.15.89:0/3545264344":"2024-02-11T06:08:38.557858+0000","172.21.15.89:6800/1397975713":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/1754666852":"2024-02-11T06:09:42.089772+0000","172.21.15.89:6800/3711070832":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/2946515553":"2024-02-11T06:08:08.869122+0000","172.21.15.89:6801/3711070832":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/2878753730":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/478488956":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/2858441641":"2024-02-11T06:08:08.869122+0000","172.21.15.89:6801/1397975713":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/632061317":"2024-02-11T06:08:38.557858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:13:06.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.278+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/2582945010 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fccf40692a0 msgr2=0x7fccf406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:06.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.278+0000 7fcd0ecb5700 1 --2- 172.21.15.89:0/2582945010 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fccf40692a0 0x7fccf406b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009fd0 tx=0x7fccf8009f20 comp rx=0 tx=0).stop 2024-02-10T06:13:06.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.278+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/2582945010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd080702d0 msgr2=0x7fcd08115ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:06.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.278+0000 7fcd0ecb5700 1 --2- 172.21.15.89:0/2582945010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd080702d0 0x7fcd08115ee0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcd00008dd0 tx=0x7fcd00004d10 comp rx=0 tx=0).stop 2024-02-10T06:13:06.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.278+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/2582945010 shutdown_connections 2024-02-10T06:13:06.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.278+0000 7fcd0ecb5700 1 --2- 172.21.15.89:0/2582945010 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fccf40692a0 0x7fccf406b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:06.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.278+0000 7fcd0ecb5700 1 --2- 172.21.15.89:0/2582945010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd080702d0 0x7fcd08115ee0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:06.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.278+0000 7fcd0ecb5700 1 --2- 172.21.15.89:0/2582945010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd08070be0 0x7fcd08116420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:06.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.278+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/2582945010 >> 172.21.15.89:0/2582945010 conn(0x7fcd080f8ae0 msgr2=0x7fcd08103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:06.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.278+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/2582945010 shutdown_connections 2024-02-10T06:13:06.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:06.278+0000 7fcd0ecb5700 1 -- 172.21.15.89:0/2582945010 wait complete. 2024-02-10T06:13:06.599 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:06 smithi089 ceph-mon[29217]: Detected new or changed devices on smithi173 2024-02-10T06:13:06.599 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:06 smithi089 ceph-mon[29217]: Adjusting osd_memory_target on smithi173 to 5312M 2024-02-10T06:13:06.599 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:06 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:13:06.599 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:06 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:13:06.599 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:06 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:06.600 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:06 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2582945010' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:13:06.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:06 smithi173 ceph-mon[36251]: Detected new or changed devices on smithi173 2024-02-10T06:13:06.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:06 smithi173 ceph-mon[36251]: Adjusting osd_memory_target on smithi173 to 5312M 2024-02-10T06:13:06.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:06 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:13:06.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:06 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:13:06.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:06 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:13:06.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:06 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2582945010' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:13:06.910 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T06:11:52.416365+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-10T06:13:06.910 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-10T06:13:07.165 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:07.571 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:07 smithi089 ceph-mon[29217]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:07.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:07 smithi173 ceph-mon[36251]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.720+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3866100809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce68102800 msgr2=0x7fce680fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.720+0000 7fce6fbfe700 1 --2- 172.21.15.89:0/3866100809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce68102800 0x7fce680fbe70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fce64004940 tx=0x7fce64009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.720+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3866100809 shutdown_connections 2024-02-10T06:13:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.720+0000 7fce6fbfe700 1 --2- 172.21.15.89:0/3866100809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce68102800 0x7fce680fbe70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.720+0000 7fce6fbfe700 1 --2- 172.21.15.89:0/3866100809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce68101e60 0x7fce68102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.720+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3866100809 >> 172.21.15.89:0/3866100809 conn(0x7fce680f7a10 msgr2=0x7fce680f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.720+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3866100809 shutdown_connections 2024-02-10T06:13:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.720+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3866100809 wait complete. 2024-02-10T06:13:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6fbfe700 1 Processor -- start 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6fbfe700 1 -- start start 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6fbfe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce68101e60 0x7fce68112a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6fbfe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce68102800 0x7fce68112f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6fbfe700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce68113590 con 0x7fce68102800 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6fbfe700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6810c8c0 con 0x7fce68101e60 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6d199700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce68102800 0x7fce68112f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6d199700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce68102800 0x7fce68112f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49178/0 (socket says 172.21.15.89:49178) 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6d199700 1 -- 172.21.15.89:0/3666705667 learned_addr learned my addr 172.21.15.89:0/3666705667 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6d99a700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce68101e60 0x7fce68112a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6d199700 1 -- 172.21.15.89:0/3666705667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce68101e60 msgr2=0x7fce68112a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6d199700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce68101e60 0x7fce68112a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:08.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6d199700 1 -- 172.21.15.89:0/3666705667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce64005040 con 0x7fce68102800 2024-02-10T06:13:08.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.721+0000 7fce6d99a700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce68101e60 0x7fce68112a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:13:08.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.722+0000 7fce6d199700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce68102800 0x7fce68112f40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fce640090d0 tx=0x7fce64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:08.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.722+0000 7fce5e7fc700 1 -- 172.21.15.89:0/3666705667 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce6400e040 con 0x7fce68102800 2024-02-10T06:13:08.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.722+0000 7fce5e7fc700 1 -- 172.21.15.89:0/3666705667 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce6400bbf0 con 0x7fce68102800 2024-02-10T06:13:08.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.722+0000 7fce5e7fc700 1 -- 172.21.15.89:0/3666705667 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce6401c630 con 0x7fce68102800 2024-02-10T06:13:08.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.722+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3666705667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6810cb40 con 0x7fce68102800 2024-02-10T06:13:08.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.722+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3666705667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6810d080 con 0x7fce68102800 2024-02-10T06:13:08.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.723+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3666705667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce68104330 con 0x7fce68102800 2024-02-10T06:13:08.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.723+0000 7fce5e7fc700 1 -- 172.21.15.89:0/3666705667 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fce6400bd60 con 0x7fce68102800 2024-02-10T06:13:08.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.723+0000 7fce5e7fc700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fce540692a0 0x7fce5406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:08.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.724+0000 7fce5e7fc700 1 -- 172.21.15.89:0/3666705667 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fce640881b0 con 0x7fce68102800 2024-02-10T06:13:08.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.724+0000 7fce6d99a700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fce540692a0 0x7fce5406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:08.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.724+0000 7fce6d99a700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fce540692a0 0x7fce5406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce580041d0 tx=0x7fce58009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:08.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.728+0000 7fce5e7fc700 1 -- 172.21.15.89:0/3666705667 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fce640582b0 con 0x7fce68102800 2024-02-10T06:13:08.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.912+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3666705667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fce68108f40 con 0x7fce68102800 2024-02-10T06:13:08.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.912+0000 7fce5e7fc700 1 -- 172.21.15.89:0/3666705667 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fce64013e60 con 0x7fce68102800 2024-02-10T06:13:08.913 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-02-10T06:13:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.914+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3666705667 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fce540692a0 msgr2=0x7fce5406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.914+0000 7fce6fbfe700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fce540692a0 0x7fce5406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce580041d0 tx=0x7fce58009040 comp rx=0 tx=0).stop 2024-02-10T06:13:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.914+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3666705667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce68102800 msgr2=0x7fce68112f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.914+0000 7fce6fbfe700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce68102800 0x7fce68112f40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fce640090d0 tx=0x7fce64009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.914+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3666705667 shutdown_connections 2024-02-10T06:13:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.914+0000 7fce6fbfe700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce68101e60 0x7fce68112a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.914+0000 7fce6fbfe700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fce540692a0 0x7fce5406b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.914+0000 7fce6fbfe700 1 --2- 172.21.15.89:0/3666705667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce68102800 0x7fce68112f40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.914+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3666705667 >> 172.21.15.89:0/3666705667 conn(0x7fce680f7a10 msgr2=0x7fce680f9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.914+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3666705667 shutdown_connections 2024-02-10T06:13:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:08.914+0000 7fce6fbfe700 1 -- 172.21.15.89:0/3666705667 wait complete. 2024-02-10T06:13:09.296 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:08 smithi089 ceph-mon[29217]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:09.296 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:08 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3666705667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T06:13:09.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:08 smithi173 ceph-mon[36251]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:09.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:08 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/3666705667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T06:13:09.549 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T06:13:09.549 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T06:13:09.802 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:11.245 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:11 smithi089 ceph-mon[29217]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:11.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.444+0000 7f0dd2b06700 1 -- 172.21.15.89:0/253194808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0dcc0fe2b0 msgr2=0x7f0dcc0fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:11.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.444+0000 7f0dd2b06700 1 --2- 172.21.15.89:0/253194808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0dcc0fe2b0 0x7f0dcc0fe700 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0db8004940 tx=0x7f0db8009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:11.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.444+0000 7f0dd2b06700 1 -- 172.21.15.89:0/253194808 shutdown_connections 2024-02-10T06:13:11.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.444+0000 7f0dd2b06700 1 --2- 172.21.15.89:0/253194808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0dcc0fe2b0 0x7f0dcc0fe700 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:11.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.444+0000 7f0dd2b06700 1 --2- 172.21.15.89:0/253194808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0dcc0fb220 0x7f0dcc0fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:11.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.444+0000 7f0dd2b06700 1 -- 172.21.15.89:0/253194808 >> 172.21.15.89:0/253194808 conn(0x7f0dcc0f6ca0 msgr2=0x7f0dcc0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:11.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.444+0000 7f0dd2b06700 1 -- 172.21.15.89:0/253194808 shutdown_connections 2024-02-10T06:13:11.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.444+0000 7f0dd2b06700 1 -- 172.21.15.89:0/253194808 wait complete. 2024-02-10T06:13:11.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd2b06700 1 Processor -- start 2024-02-10T06:13:11.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd2b06700 1 -- start start 2024-02-10T06:13:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd2b06700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0dcc0fb220 0x7f0dcc10da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd2b06700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0dcc0fe2b0 0x7f0dcc10dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd2b06700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc10e5d0 con 0x7f0dcc0fb220 2024-02-10T06:13:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd2b06700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc10ee20 con 0x7f0dcc0fe2b0 2024-02-10T06:13:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd08a2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0dcc0fb220 0x7f0dcc10da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd08a2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0dcc0fb220 0x7f0dcc10da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49206/0 (socket says 172.21.15.89:49206) 2024-02-10T06:13:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd08a2700 1 -- 172.21.15.89:0/3696558649 learned_addr learned my addr 172.21.15.89:0/3696558649 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:11.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dcbfff700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0dcc0fe2b0 0x7f0dcc10dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd08a2700 1 -- 172.21.15.89:0/3696558649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0dcc0fe2b0 msgr2=0x7f0dcc10dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd08a2700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0dcc0fe2b0 0x7f0dcc10dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dd08a2700 1 -- 172.21.15.89:0/3696558649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db8005040 con 0x7f0dcc0fb220 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.445+0000 7f0dcbfff700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0dcc0fe2b0 0x7f0dcc10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.446+0000 7f0dd08a2700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0dcc0fb220 0x7f0dcc10da90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0002a50 tx=0x7f0dc0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.446+0000 7f0dc97fa700 1 -- 172.21.15.89:0/3696558649 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dc0007cc0 con 0x7f0dcc0fb220 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.446+0000 7f0dd2b06700 1 -- 172.21.15.89:0/3696558649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dcc10f100 con 0x7f0dcc0fb220 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.446+0000 7f0dc97fa700 1 -- 172.21.15.89:0/3696558649 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0dc000f040 con 0x7f0dcc0fb220 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.446+0000 7f0dc97fa700 1 -- 172.21.15.89:0/3696558649 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dc000da40 con 0x7f0dcc0fb220 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.446+0000 7f0dd2b06700 1 -- 172.21.15.89:0/3696558649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dcc10f640 con 0x7f0dcc0fb220 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.447+0000 7f0dd2b06700 1 -- 172.21.15.89:0/3696558649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dcc107aa0 con 0x7f0dcc0fb220 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.447+0000 7f0dc97fa700 1 -- 172.21.15.89:0/3696558649 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0dc00097c0 con 0x7f0dcc0fb220 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.447+0000 7f0dc97fa700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0dbc0692a0 0x7f0dbc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.448+0000 7f0dc97fa700 1 -- 172.21.15.89:0/3696558649 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0dc0086bd0 con 0x7f0dcc0fb220 2024-02-10T06:13:11.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.448+0000 7f0dcbfff700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0dbc0692a0 0x7f0dbc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:11.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.448+0000 7f0dcbfff700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0dbc0692a0 0x7f0dbc06b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0db8004f90 tx=0x7f0db8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:11.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.452+0000 7f0dc97fa700 1 -- 172.21.15.89:0/3696558649 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0dc00569c0 con 0x7f0dcc0fb220 2024-02-10T06:13:11.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:11 smithi173 ceph-mon[36251]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:11.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.712+0000 7f0dd2b06700 1 -- 172.21.15.89:0/3696558649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f0dcc04bed0 con 0x7f0dcc0fb220 2024-02-10T06:13:11.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.716+0000 7f0dc97fa700 1 -- 172.21.15.89:0/3696558649 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f0dc00567e0 con 0x7f0dcc0fb220 2024-02-10T06:13:11.717 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2024-02-10T06:13:11.717 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQD3E8dl57qGKhAAvj2rQnukNXlNSKkm6+Ho/A== 2024-02-10T06:13:11.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.718+0000 7f0dd2b06700 1 -- 172.21.15.89:0/3696558649 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0dbc0692a0 msgr2=0x7f0dbc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:11.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.718+0000 7f0dd2b06700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0dbc0692a0 0x7f0dbc06b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0db8004f90 tx=0x7f0db8009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:11.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.718+0000 7f0dd2b06700 1 -- 172.21.15.89:0/3696558649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0dcc0fb220 msgr2=0x7f0dcc10da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:11.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.718+0000 7f0dd2b06700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0dcc0fb220 0x7f0dcc10da90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0002a50 tx=0x7f0dc0004600 comp rx=0 tx=0).stop 2024-02-10T06:13:11.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.718+0000 7f0dd2b06700 1 -- 172.21.15.89:0/3696558649 shutdown_connections 2024-02-10T06:13:11.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.718+0000 7f0dd2b06700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0dbc0692a0 0x7f0dbc06b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:11.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.718+0000 7f0dd2b06700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0dcc0fb220 0x7f0dcc10da90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:11.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.718+0000 7f0dd2b06700 1 --2- 172.21.15.89:0/3696558649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0dcc0fe2b0 0x7f0dcc10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:11.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.718+0000 7f0dd2b06700 1 -- 172.21.15.89:0/3696558649 >> 172.21.15.89:0/3696558649 conn(0x7f0dcc0f6ca0 msgr2=0x7f0dcc0f8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:11.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.718+0000 7f0dd2b06700 1 -- 172.21.15.89:0/3696558649 shutdown_connections 2024-02-10T06:13:11.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:11.718+0000 7f0dd2b06700 1 -- 172.21.15.89:0/3696558649 wait complete. 2024-02-10T06:13:12.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:12 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3696558649' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:13:12.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:12 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3696558649' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:13:12.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:12.501 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:13:12.501 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T06:13:12.501 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T06:13:12.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T06:13:12.575 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:12 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/3696558649' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:13:12.575 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:12 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/3696558649' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:13:12.576 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:12.809 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi173/config 2024-02-10T06:13:13.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:13 smithi089 ceph-mon[29217]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:13.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:13 smithi173 ceph-mon[36251]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:14.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.334+0000 7f488d72d700 1 -- 172.21.15.173:0/1704345409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48880fda80 msgr2=0x7f48880ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:14.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.334+0000 7f488d72d700 1 --2- 172.21.15.173:0/1704345409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48880fda80 0x7f48880ffe60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4878004940 tx=0x7f4878009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:14.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.334+0000 7f488d72d700 1 -- 172.21.15.173:0/1704345409 shutdown_connections 2024-02-10T06:13:14.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.334+0000 7f488d72d700 1 --2- 172.21.15.173:0/1704345409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48881003a0 0x7f4888102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:14.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.334+0000 7f488d72d700 1 --2- 172.21.15.173:0/1704345409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48880fda80 0x7f48880ffe60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:14.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.334+0000 7f488d72d700 1 -- 172.21.15.173:0/1704345409 >> 172.21.15.173:0/1704345409 conn(0x7f48880f74e0 msgr2=0x7f48880f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.335+0000 7f488d72d700 1 -- 172.21.15.173:0/1704345409 shutdown_connections 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.335+0000 7f488d72d700 1 -- 172.21.15.173:0/1704345409 wait complete. 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.335+0000 7f488d72d700 1 Processor -- start 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.335+0000 7f488d72d700 1 -- start start 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.335+0000 7f488d72d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880fda80 0x7f48880fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.335+0000 7f488d72d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48881003a0 0x7f48880fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.335+0000 7f488d72d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48880fbf90 con 0x7f48880fda80 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.335+0000 7f488d72d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48880fc100 con 0x7f48881003a0 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.335+0000 7f487e7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48881003a0 0x7f48880fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f487e7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48881003a0 0x7f48880fb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:59720/0 (socket says 172.21.15.173:59720) 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f487e7fc700 1 -- 172.21.15.173:0/1763881389 learned_addr learned my addr 172.21.15.173:0/1763881389 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-10T06:13:14.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f4886ffd700 1 --2- 172.21.15.173:0/1763881389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880fda80 0x7f48880fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:14.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f487e7fc700 1 -- 172.21.15.173:0/1763881389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880fda80 msgr2=0x7f48880fd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:14.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f487e7fc700 1 --2- 172.21.15.173:0/1763881389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880fda80 0x7f48880fd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:14.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f487e7fc700 1 -- 172.21.15.173:0/1763881389 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4878005040 con 0x7f48881003a0 2024-02-10T06:13:14.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f487e7fc700 1 --2- 172.21.15.173:0/1763881389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48881003a0 0x7f48880fb990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4870002a50 tx=0x7f48700046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:14.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f487ffff700 1 -- 172.21.15.173:0/1763881389 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4870007cc0 con 0x7f48881003a0 2024-02-10T06:13:14.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f487ffff700 1 -- 172.21.15.173:0/1763881389 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f487000f040 con 0x7f48881003a0 2024-02-10T06:13:14.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f488d72d700 1 -- 172.21.15.173:0/1763881389 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48880fc3e0 con 0x7f48881003a0 2024-02-10T06:13:14.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f487ffff700 1 -- 172.21.15.173:0/1763881389 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487000da30 con 0x7f48881003a0 2024-02-10T06:13:14.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.336+0000 7f488d72d700 1 -- 172.21.15.173:0/1763881389 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4888115970 con 0x7f48881003a0 2024-02-10T06:13:14.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.337+0000 7f488d72d700 1 -- 172.21.15.173:0/1763881389 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488804bed0 con 0x7f48881003a0 2024-02-10T06:13:14.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.337+0000 7f487ffff700 1 -- 172.21.15.173:0/1763881389 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f487000db90 con 0x7f48881003a0 2024-02-10T06:13:14.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.338+0000 7f487ffff700 1 --2- 172.21.15.173:0/1763881389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f48740692a0 0x7f487406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:14.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.338+0000 7f487ffff700 1 -- 172.21.15.173:0/1763881389 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4870086be0 con 0x7f48881003a0 2024-02-10T06:13:14.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.338+0000 7f4886ffd700 1 --2- 172.21.15.173:0/1763881389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f48740692a0 0x7f487406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:14.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.339+0000 7f4886ffd700 1 --2- 172.21.15.173:0/1763881389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f48740692a0 0x7f487406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4878004890 tx=0x7f4878009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:14.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.342+0000 7f487ffff700 1 -- 172.21.15.173:0/1763881389 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4870056c60 con 0x7f48881003a0 2024-02-10T06:13:14.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.594+0000 7f488d72d700 1 -- 172.21.15.173:0/1763881389 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f4888105a10 con 0x7f48881003a0 2024-02-10T06:13:14.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.602+0000 7f487ffff700 1 -- 172.21.15.173:0/1763881389 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f48700567f0 con 0x7f48881003a0 2024-02-10T06:13:14.602 INFO:teuthology.orchestra.run.smithi173.stdout:[client.1] 2024-02-10T06:13:14.602 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQD6E8dla7COIxAALkXqTwIu1F3FILO3+2Us+A== 2024-02-10T06:13:14.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.604+0000 7f488d72d700 1 -- 172.21.15.173:0/1763881389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f48740692a0 msgr2=0x7f487406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:14.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.604+0000 7f488d72d700 1 --2- 172.21.15.173:0/1763881389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f48740692a0 0x7f487406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4878004890 tx=0x7f4878009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:14.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.604+0000 7f488d72d700 1 -- 172.21.15.173:0/1763881389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48881003a0 msgr2=0x7f48880fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:14.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.604+0000 7f488d72d700 1 --2- 172.21.15.173:0/1763881389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48881003a0 0x7f48880fb990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4870002a50 tx=0x7f48700046b0 comp rx=0 tx=0).stop 2024-02-10T06:13:14.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.604+0000 7f488d72d700 1 -- 172.21.15.173:0/1763881389 shutdown_connections 2024-02-10T06:13:14.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.604+0000 7f488d72d700 1 --2- 172.21.15.173:0/1763881389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f48740692a0 0x7f487406b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:14.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.604+0000 7f488d72d700 1 --2- 172.21.15.173:0/1763881389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48880fda80 0x7f48880fd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:14.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.604+0000 7f488d72d700 1 --2- 172.21.15.173:0/1763881389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48881003a0 0x7f48880fb990 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:14.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.604+0000 7f488d72d700 1 -- 172.21.15.173:0/1763881389 >> 172.21.15.173:0/1763881389 conn(0x7f48880f74e0 msgr2=0x7f48880f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:14.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.604+0000 7f488d72d700 1 -- 172.21.15.173:0/1763881389 shutdown_connections 2024-02-10T06:13:14.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-10T06:13:14.604+0000 7f488d72d700 1 -- 172.21.15.173:0/1763881389 wait complete. 2024-02-10T06:13:15.180 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:13:15.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-10T06:13:15.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-10T06:13:15.262 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T06:13:15.262 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T06:13:15.263 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T06:13:15.291 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:15 smithi089 ceph-mon[29217]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:15.291 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:15 smithi089 ceph-mon[29217]: from='client.? 172.21.15.173:0/1763881389' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:13:15.291 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:15 smithi089 ceph-mon[29217]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:13:15.291 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:15 smithi089 ceph-mon[29217]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:13:15.530 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:15.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:15 smithi173 ceph-mon[36251]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:15.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:15 smithi173 ceph-mon[36251]: from='client.? 172.21.15.173:0/1763881389' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:13:15.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:15 smithi173 ceph-mon[36251]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T06:13:15.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:15 smithi173 ceph-mon[36251]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T06:13:17.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.135+0000 7f49b51cc700 1 -- 172.21.15.89:0/3732979092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49b0102f50 msgr2=0x7f49b0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:17.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.135+0000 7f49b51cc700 1 --2- 172.21.15.89:0/3732979092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49b0102f50 0x7f49b0103320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f49a0004940 tx=0x7f49a0009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:17.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.135+0000 7f49b51cc700 1 -- 172.21.15.89:0/3732979092 shutdown_connections 2024-02-10T06:13:17.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.135+0000 7f49b51cc700 1 --2- 172.21.15.89:0/3732979092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49b01038f0 0x7f49b00fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:17.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.135+0000 7f49b51cc700 1 --2- 172.21.15.89:0/3732979092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49b0102f50 0x7f49b0103320 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.135+0000 7f49b51cc700 1 -- 172.21.15.89:0/3732979092 >> 172.21.15.89:0/3732979092 conn(0x7f49b00f8ae0 msgr2=0x7f49b00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.135+0000 7f49b51cc700 1 -- 172.21.15.89:0/3732979092 shutdown_connections 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.135+0000 7f49b51cc700 1 -- 172.21.15.89:0/3732979092 wait complete. 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.135+0000 7f49b51cc700 1 Processor -- start 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.135+0000 7f49b51cc700 1 -- start start 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49b51cc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49b0102f50 0x7f49b0113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49b51cc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49b01038f0 0x7f49b0114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49b51cc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b0114700 con 0x7f49b01038f0 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49b51cc700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b010d9a0 con 0x7f49b0102f50 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49a7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49b01038f0 0x7f49b0114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49a7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49b01038f0 0x7f49b0114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49224/0 (socket says 172.21.15.89:49224) 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49a7fff700 1 -- 172.21.15.89:0/1141736389 learned_addr learned my addr 172.21.15.89:0/1141736389 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49a7fff700 1 -- 172.21.15.89:0/1141736389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49b0102f50 msgr2=0x7f49b0113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49a7fff700 1 --2- 172.21.15.89:0/1141736389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49b0102f50 0x7f49b0113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:17.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49a7fff700 1 -- 172.21.15.89:0/1141736389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a0005040 con 0x7f49b01038f0 2024-02-10T06:13:17.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49a7fff700 1 --2- 172.21.15.89:0/1141736389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49b01038f0 0x7f49b0114020 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f499800b790 tx=0x7f4998009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:17.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49a77fe700 1 -- 172.21.15.89:0/1141736389 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f499800f6f0 con 0x7f49b01038f0 2024-02-10T06:13:17.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49a77fe700 1 -- 172.21.15.89:0/1141736389 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f499800fd20 con 0x7f49b01038f0 2024-02-10T06:13:17.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49b51cc700 1 -- 172.21.15.89:0/1141736389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b010dc80 con 0x7f49b01038f0 2024-02-10T06:13:17.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49a77fe700 1 -- 172.21.15.89:0/1141736389 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49980103a0 con 0x7f49b01038f0 2024-02-10T06:13:17.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.136+0000 7f49b51cc700 1 -- 172.21.15.89:0/1141736389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b010e1c0 con 0x7f49b01038f0 2024-02-10T06:13:17.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.137+0000 7f49b51cc700 1 -- 172.21.15.89:0/1141736389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b00fc6e0 con 0x7f49b01038f0 2024-02-10T06:13:17.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.138+0000 7f49a77fe700 1 -- 172.21.15.89:0/1141736389 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4998011400 con 0x7f49b01038f0 2024-02-10T06:13:17.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.138+0000 7f49a77fe700 1 --2- 172.21.15.89:0/1141736389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f499c0692a0 0x7f499c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:17.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.138+0000 7f49aed9d700 1 --2- 172.21.15.89:0/1141736389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f499c0692a0 0x7f499c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:17.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.138+0000 7f49a77fe700 1 -- 172.21.15.89:0/1141736389 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4998086d10 con 0x7f49b01038f0 2024-02-10T06:13:17.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.138+0000 7f49aed9d700 1 --2- 172.21.15.89:0/1141736389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f499c0692a0 0x7f499c06b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f49a0004f90 tx=0x7f49a0009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:17.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.142+0000 7f49a77fe700 1 -- 172.21.15.89:0/1141736389 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4998056d90 con 0x7f49b01038f0 2024-02-10T06:13:17.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.378+0000 7f49b51cc700 1 -- 172.21.15.89:0/1141736389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f49b004bed0 con 0x7f49b01038f0 2024-02-10T06:13:17.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.382+0000 7f49a77fe700 1 -- 172.21.15.89:0/1141736389 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167168 (secure 0 0 0) 0x7f4998056920 con 0x7f49b01038f0 2024-02-10T06:13:17.385 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:13:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.384+0000 7f49b51cc700 1 -- 172.21.15.89:0/1141736389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f499c0692a0 msgr2=0x7f499c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.384+0000 7f49b51cc700 1 --2- 172.21.15.89:0/1141736389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f499c0692a0 0x7f499c06b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f49a0004f90 tx=0x7f49a0009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.384+0000 7f49b51cc700 1 -- 172.21.15.89:0/1141736389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49b01038f0 msgr2=0x7f49b0114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:17.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.384+0000 7f49b51cc700 1 --2- 172.21.15.89:0/1141736389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49b01038f0 0x7f49b0114020 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f499800b790 tx=0x7f4998009220 comp rx=0 tx=0).stop 2024-02-10T06:13:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.384+0000 7f49b51cc700 1 -- 172.21.15.89:0/1141736389 shutdown_connections 2024-02-10T06:13:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.384+0000 7f49b51cc700 1 --2- 172.21.15.89:0/1141736389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49b0102f50 0x7f49b0113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.384+0000 7f49b51cc700 1 --2- 172.21.15.89:0/1141736389 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f499c0692a0 0x7f499c06b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.384+0000 7f49b51cc700 1 --2- 172.21.15.89:0/1141736389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49b01038f0 0x7f49b0114020 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.384+0000 7f49b51cc700 1 -- 172.21.15.89:0/1141736389 >> 172.21.15.89:0/1141736389 conn(0x7f49b00f8ae0 msgr2=0x7f49b00fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.384+0000 7f49b51cc700 1 -- 172.21.15.89:0/1141736389 shutdown_connections 2024-02-10T06:13:17.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:17.384+0000 7f49b51cc700 1 -- 172.21.15.89:0/1141736389 wait complete. 2024-02-10T06:13:17.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:17 smithi089 ceph-mon[29217]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:17.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:17 smithi173 ceph-mon[36251]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:18.326 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":19,"active_gid":14210,"active_name":"smithi089.pwmrye","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":223399833},{"type":"v1","addr":"172.21.15.89:6801","nonce":223399833}]},"active_addr":"172.21.15.89:6801/223399833","active_change":"2024-02-10T06:09:42.089930+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi173.swdirm","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.89:8443/","prometheus":"http://172.21.15.89:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3550045642}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3648810985}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3615761864}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3505016878}]}]} 2024-02-10T06:13:18.330 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T06:13:18.330 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T06:13:18.330 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T06:13:18.399 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:18 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/1141736389' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:13:18.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:18 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1141736389' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T06:13:18.622 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:19.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:19 smithi089 ceph-mon[29217]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:19.597 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:19 smithi173 ceph-mon[36251]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.216+0000 7f9d74539700 1 -- 172.21.15.89:0/1626427364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d6c101e60 msgr2=0x7f9d6c102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.216+0000 7f9d74539700 1 --2- 172.21.15.89:0/1626427364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d6c101e60 0x7f9d6c102230 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9d68004940 tx=0x7f9d68009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.216+0000 7f9d74539700 1 -- 172.21.15.89:0/1626427364 shutdown_connections 2024-02-10T06:13:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.216+0000 7f9d74539700 1 --2- 172.21.15.89:0/1626427364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c102800 0x7f9d6c0fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.216+0000 7f9d74539700 1 --2- 172.21.15.89:0/1626427364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d6c101e60 0x7f9d6c102230 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.216+0000 7f9d74539700 1 -- 172.21.15.89:0/1626427364 >> 172.21.15.89:0/1626427364 conn(0x7f9d6c0f7a10 msgr2=0x7f9d6c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.216+0000 7f9d74539700 1 -- 172.21.15.89:0/1626427364 shutdown_connections 2024-02-10T06:13:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d74539700 1 -- 172.21.15.89:0/1626427364 wait complete. 2024-02-10T06:13:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d74539700 1 Processor -- start 2024-02-10T06:13:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d74539700 1 -- start start 2024-02-10T06:13:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d74539700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c101e60 0x7f9d6c113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d74539700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d6c102800 0x7f9d6c114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d74539700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c114680 con 0x7f9d6c102800 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d74539700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c10d9b0 con 0x7f9d6c101e60 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d71ad4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d6c102800 0x7f9d6c114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d71ad4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d6c102800 0x7f9d6c114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54448/0 (socket says 172.21.15.89:54448) 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d71ad4700 1 -- 172.21.15.89:0/242239018 learned_addr learned my addr 172.21.15.89:0/242239018 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d722d5700 1 --2- 172.21.15.89:0/242239018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c101e60 0x7f9d6c113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d71ad4700 1 -- 172.21.15.89:0/242239018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c101e60 msgr2=0x7f9d6c113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.217+0000 7f9d71ad4700 1 --2- 172.21.15.89:0/242239018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c101e60 0x7f9d6c113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.218+0000 7f9d71ad4700 1 -- 172.21.15.89:0/242239018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d68005040 con 0x7f9d6c102800 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.218+0000 7f9d71ad4700 1 --2- 172.21.15.89:0/242239018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d6c102800 0x7f9d6c114030 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c002a50 tx=0x7f9d5c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.218+0000 7f9d62ffd700 1 -- 172.21.15.89:0/242239018 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d5c007cc0 con 0x7f9d6c102800 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.218+0000 7f9d62ffd700 1 -- 172.21.15.89:0/242239018 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d5c00f040 con 0x7f9d6c102800 2024-02-10T06:13:20.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.218+0000 7f9d62ffd700 1 -- 172.21.15.89:0/242239018 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d5c00d9c0 con 0x7f9d6c102800 2024-02-10T06:13:20.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.218+0000 7f9d74539700 1 -- 172.21.15.89:0/242239018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6c10dc90 con 0x7f9d6c102800 2024-02-10T06:13:20.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.218+0000 7f9d74539700 1 -- 172.21.15.89:0/242239018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6c10e1d0 con 0x7f9d6c102800 2024-02-10T06:13:20.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.219+0000 7f9d74539700 1 -- 172.21.15.89:0/242239018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6c0ff170 con 0x7f9d6c102800 2024-02-10T06:13:20.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.219+0000 7f9d62ffd700 1 -- 172.21.15.89:0/242239018 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9d5c009730 con 0x7f9d6c102800 2024-02-10T06:13:20.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.220+0000 7f9d62ffd700 1 --2- 172.21.15.89:0/242239018 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d580692a0 0x7f9d5806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:20.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.220+0000 7f9d62ffd700 1 -- 172.21.15.89:0/242239018 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9d5c086c20 con 0x7f9d6c102800 2024-02-10T06:13:20.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.220+0000 7f9d722d5700 1 --2- 172.21.15.89:0/242239018 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d580692a0 0x7f9d5806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:20.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.220+0000 7f9d722d5700 1 --2- 172.21.15.89:0/242239018 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d580692a0 0x7f9d5806b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9d68004f90 tx=0x7f9d68009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:20.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.224+0000 7f9d62ffd700 1 -- 172.21.15.89:0/242239018 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9d5c056c40 con 0x7f9d6c102800 2024-02-10T06:13:20.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.406+0000 7f9d74539700 1 -- 172.21.15.89:0/242239018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d6c10a050 con 0x7f9d6c102800 2024-02-10T06:13:20.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.407+0000 7f9d62ffd700 1 -- 172.21.15.89:0/242239018 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11058 (secure 0 0 0) 0x7f9d5c0567d0 con 0x7f9d6c102800 2024-02-10T06:13:20.408 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:13:20.408 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":41,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","created":"2024-02-10T06:07:41.156060+0000","modified":"2024-02-10T06:13:04.323413+0000","last_up_change":"2024-02-10T06:13:03.312644+0000","last_in_change":"2024-02-10T06:12:45.108728+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T06:11:52.416365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fae919bb-103f-402d-8509-1e29a0ebc746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6803","nonce":2704721040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6805","nonce":2704721040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6809","nonce":2704721040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6807","nonce":2704721040}]},"public_addr":"172.21.15.89:6803/2704721040","cluster_addr":"172.21.15.89:6805/2704721040","heartbeat_back_addr":"172.21.15.89:6809/2704721040","heartbeat_front_addr":"172.21.15.89:6807/2704721040","state":["exists","up"]},{"osd":1,"uuid":"61122d5c-670d-4d4f-8575-d3759758e348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6811","nonce":3051839828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6813","nonce":3051839828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6817","nonce":3051839828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6815","nonce":3051839828}]},"public_addr":"172.21.15.89:6811/3051839828","cluster_addr":"172.21.15.89:6813/3051839828","heartbeat_back_addr":"172.21.15.89:6817/3051839828","heartbeat_front_addr":"172.21.15.89:6815/3051839828","state":["exists","up"]},{"osd":2,"uuid":"3ffd67bc-829e-449f-b4c3-94a11da55da6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6819","nonce":637372379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6821","nonce":637372379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6825","nonce":637372379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6823","nonce":637372379}]},"public_addr":"172.21.15.89:6819/637372379","cluster_addr":"172.21.15.89:6821/637372379","heartbeat_back_addr":"172.21.15.89:6825/637372379","heartbeat_front_addr":"172.21.15.89:6823/637372379","state":["exists","up"]},{"osd":3,"uuid":"cc150631-29f2-4b5d-846d-bffbe2c09cec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6801","nonce":2644357236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6803","nonce":2644357236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6807","nonce":2644357236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6805","nonce":2644357236}]},"public_addr":"172.21.15.173:6801/2644357236","cluster_addr":"172.21.15.173:6803/2644357236","heartbeat_back_addr":"172.21.15.173:6807/2644357236","heartbeat_front_addr":"172.21.15.173:6805/2644357236","state":["exists","up"]},{"osd":4,"uuid":"cccd0c3a-15fc-43e4-852d-f392f991ef61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6809","nonce":2725914780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6811","nonce":2725914780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6815","nonce":2725914780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6813","nonce":2725914780}]},"public_addr":"172.21.15.173:6809/2725914780","cluster_addr":"172.21.15.173:6811/2725914780","heartbeat_back_addr":"172.21.15.173:6815/2725914780","heartbeat_front_addr":"172.21.15.173:6813/2725914780","state":["exists","up"]},{"osd":5,"uuid":"00386df2-a6d8-42e8-adfe-58b6141471f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6817","nonce":49288782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6819","nonce":49288782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6823","nonce":49288782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6821","nonce":49288782}]},"public_addr":"172.21.15.173:6817/49288782","cluster_addr":"172.21.15.173:6819/49288782","heartbeat_back_addr":"172.21.15.173:6823/49288782","heartbeat_front_addr":"172.21.15.173:6821/49288782","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:10:55.172572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:11:22.140314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:11:49.151682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:12:12.352626+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:12:37.457216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:13:02.197643+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/1955086590":"2024-02-11T06:09:42.089772+0000","172.21.15.89:0/644693661":"2024-02-11T06:09:42.089772+0000","172.21.15.89:0/1015341098":"2024-02-11T06:09:42.089772+0000","172.21.15.89:6800/1955086590":"2024-02-11T06:09:42.089772+0000","172.21.15.89:0/3545264344":"2024-02-11T06:08:38.557858+0000","172.21.15.89:6800/1397975713":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/1754666852":"2024-02-11T06:09:42.089772+0000","172.21.15.89:6800/3711070832":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/2946515553":"2024-02-11T06:08:08.869122+0000","172.21.15.89:6801/3711070832":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/2878753730":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/478488956":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/2858441641":"2024-02-11T06:08:08.869122+0000","172.21.15.89:6801/1397975713":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/632061317":"2024-02-11T06:08:38.557858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:13:20.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.409+0000 7f9d74539700 1 -- 172.21.15.89:0/242239018 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d580692a0 msgr2=0x7f9d5806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:20.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.409+0000 7f9d74539700 1 --2- 172.21.15.89:0/242239018 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d580692a0 0x7f9d5806b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9d68004f90 tx=0x7f9d68009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:20.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.409+0000 7f9d74539700 1 -- 172.21.15.89:0/242239018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d6c102800 msgr2=0x7f9d6c114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:20.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.409+0000 7f9d74539700 1 --2- 172.21.15.89:0/242239018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d6c102800 0x7f9d6c114030 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c002a50 tx=0x7f9d5c004600 comp rx=0 tx=0).stop 2024-02-10T06:13:20.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.409+0000 7f9d74539700 1 -- 172.21.15.89:0/242239018 shutdown_connections 2024-02-10T06:13:20.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.409+0000 7f9d74539700 1 --2- 172.21.15.89:0/242239018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d6c101e60 0x7f9d6c113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:20.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.409+0000 7f9d74539700 1 --2- 172.21.15.89:0/242239018 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d580692a0 0x7f9d5806b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:20.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.409+0000 7f9d74539700 1 --2- 172.21.15.89:0/242239018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d6c102800 0x7f9d6c114030 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:20.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.409+0000 7f9d74539700 1 -- 172.21.15.89:0/242239018 >> 172.21.15.89:0/242239018 conn(0x7f9d6c0f7a10 msgr2=0x7f9d6c0f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:20.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.409+0000 7f9d74539700 1 -- 172.21.15.89:0/242239018 shutdown_connections 2024-02-10T06:13:20.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:20.409+0000 7f9d74539700 1 -- 172.21.15.89:0/242239018 wait complete. 2024-02-10T06:13:21.066 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T06:13:21.066 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T06:13:21.319 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:21.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:21 smithi089 ceph-mon[29217]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:21.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:21 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/242239018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:13:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:21 smithi173 ceph-mon[36251]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:21 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/242239018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:13:23.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.042+0000 7f864236c700 1 -- 172.21.15.89:0/2477248437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863c102800 msgr2=0x7f863c0fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:23.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.042+0000 7f864236c700 1 --2- 172.21.15.89:0/2477248437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863c102800 0x7f863c0fbe70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8630004940 tx=0x7f8630009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:23.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.042+0000 7f864236c700 1 -- 172.21.15.89:0/2477248437 shutdown_connections 2024-02-10T06:13:23.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.042+0000 7f864236c700 1 --2- 172.21.15.89:0/2477248437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863c102800 0x7f863c0fbe70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:23.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.042+0000 7f864236c700 1 --2- 172.21.15.89:0/2477248437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f863c101e60 0x7f863c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:23.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.042+0000 7f864236c700 1 -- 172.21.15.89:0/2477248437 >> 172.21.15.89:0/2477248437 conn(0x7f863c0f7a10 msgr2=0x7f863c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:23.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.042+0000 7f864236c700 1 -- 172.21.15.89:0/2477248437 shutdown_connections 2024-02-10T06:13:23.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.042+0000 7f864236c700 1 -- 172.21.15.89:0/2477248437 wait complete. 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.043+0000 7f864236c700 1 Processor -- start 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.043+0000 7f864236c700 1 -- start start 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.043+0000 7f864236c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f863c101e60 0x7f863c113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.043+0000 7f864236c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863c102800 0x7f863c114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.043+0000 7f864236c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c114670 con 0x7f863c102800 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.043+0000 7f864236c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c10d9a0 con 0x7f863c101e60 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.043+0000 7f863b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863c102800 0x7f863c114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f863b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863c102800 0x7f863c114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54468/0 (socket says 172.21.15.89:54468) 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f863b7fe700 1 -- 172.21.15.89:0/278553345 learned_addr learned my addr 172.21.15.89:0/278553345 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f863bfff700 1 --2- 172.21.15.89:0/278553345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f863c101e60 0x7f863c113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f863b7fe700 1 -- 172.21.15.89:0/278553345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f863c101e60 msgr2=0x7f863c113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f863b7fe700 1 --2- 172.21.15.89:0/278553345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f863c101e60 0x7f863c113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f863b7fe700 1 -- 172.21.15.89:0/278553345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8630005040 con 0x7f863c102800 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f863b7fe700 1 --2- 172.21.15.89:0/278553345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863c102800 0x7f863c114020 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f86300090d0 tx=0x7f8630009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:23.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f8638ff9700 1 -- 172.21.15.89:0/278553345 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f863000e040 con 0x7f863c102800 2024-02-10T06:13:23.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f8638ff9700 1 -- 172.21.15.89:0/278553345 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8630009430 con 0x7f863c102800 2024-02-10T06:13:23.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f8638ff9700 1 -- 172.21.15.89:0/278553345 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f863001c3f0 con 0x7f863c102800 2024-02-10T06:13:23.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f864236c700 1 -- 172.21.15.89:0/278553345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863c10dc20 con 0x7f863c102800 2024-02-10T06:13:23.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.044+0000 7f864236c700 1 -- 172.21.15.89:0/278553345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863c10e160 con 0x7f863c102800 2024-02-10T06:13:23.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.045+0000 7f864236c700 1 -- 172.21.15.89:0/278553345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863c0fb610 con 0x7f863c102800 2024-02-10T06:13:23.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.046+0000 7f8638ff9700 1 -- 172.21.15.89:0/278553345 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f863001c6d0 con 0x7f863c102800 2024-02-10T06:13:23.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.046+0000 7f8638ff9700 1 --2- 172.21.15.89:0/278553345 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f86240692a0 0x7f862406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:23.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.046+0000 7f8638ff9700 1 -- 172.21.15.89:0/278553345 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f86300880f0 con 0x7f863c102800 2024-02-10T06:13:23.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.046+0000 7f863bfff700 1 --2- 172.21.15.89:0/278553345 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f86240692a0 0x7f862406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:23.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.046+0000 7f863bfff700 1 --2- 172.21.15.89:0/278553345 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f86240692a0 0x7f862406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f862c0041d0 tx=0x7f862c0043c0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.050+0000 7f8638ff9700 1 -- 172.21.15.89:0/278553345 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8630058260 con 0x7f863c102800 2024-02-10T06:13:23.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.235+0000 7f864236c700 1 -- 172.21.15.89:0/278553345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f863c04bed0 con 0x7f863c102800 2024-02-10T06:13:23.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.235+0000 7f8638ff9700 1 -- 172.21.15.89:0/278553345 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11058 (secure 0 0 0) 0x7f8630086e60 con 0x7f863c102800 2024-02-10T06:13:23.301 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:13:23.301 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":41,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","created":"2024-02-10T06:07:41.156060+0000","modified":"2024-02-10T06:13:04.323413+0000","last_up_change":"2024-02-10T06:13:03.312644+0000","last_in_change":"2024-02-10T06:12:45.108728+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T06:11:52.416365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fae919bb-103f-402d-8509-1e29a0ebc746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6803","nonce":2704721040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6805","nonce":2704721040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6809","nonce":2704721040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6807","nonce":2704721040}]},"public_addr":"172.21.15.89:6803/2704721040","cluster_addr":"172.21.15.89:6805/2704721040","heartbeat_back_addr":"172.21.15.89:6809/2704721040","heartbeat_front_addr":"172.21.15.89:6807/2704721040","state":["exists","up"]},{"osd":1,"uuid":"61122d5c-670d-4d4f-8575-d3759758e348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6811","nonce":3051839828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6813","nonce":3051839828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6817","nonce":3051839828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6815","nonce":3051839828}]},"public_addr":"172.21.15.89:6811/3051839828","cluster_addr":"172.21.15.89:6813/3051839828","heartbeat_back_addr":"172.21.15.89:6817/3051839828","heartbeat_front_addr":"172.21.15.89:6815/3051839828","state":["exists","up"]},{"osd":2,"uuid":"3ffd67bc-829e-449f-b4c3-94a11da55da6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6819","nonce":637372379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6821","nonce":637372379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6825","nonce":637372379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6823","nonce":637372379}]},"public_addr":"172.21.15.89:6819/637372379","cluster_addr":"172.21.15.89:6821/637372379","heartbeat_back_addr":"172.21.15.89:6825/637372379","heartbeat_front_addr":"172.21.15.89:6823/637372379","state":["exists","up"]},{"osd":3,"uuid":"cc150631-29f2-4b5d-846d-bffbe2c09cec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6801","nonce":2644357236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6803","nonce":2644357236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6807","nonce":2644357236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6805","nonce":2644357236}]},"public_addr":"172.21.15.173:6801/2644357236","cluster_addr":"172.21.15.173:6803/2644357236","heartbeat_back_addr":"172.21.15.173:6807/2644357236","heartbeat_front_addr":"172.21.15.173:6805/2644357236","state":["exists","up"]},{"osd":4,"uuid":"cccd0c3a-15fc-43e4-852d-f392f991ef61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6809","nonce":2725914780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6811","nonce":2725914780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6815","nonce":2725914780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6813","nonce":2725914780}]},"public_addr":"172.21.15.173:6809/2725914780","cluster_addr":"172.21.15.173:6811/2725914780","heartbeat_back_addr":"172.21.15.173:6815/2725914780","heartbeat_front_addr":"172.21.15.173:6813/2725914780","state":["exists","up"]},{"osd":5,"uuid":"00386df2-a6d8-42e8-adfe-58b6141471f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6817","nonce":49288782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6819","nonce":49288782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6823","nonce":49288782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6821","nonce":49288782}]},"public_addr":"172.21.15.173:6817/49288782","cluster_addr":"172.21.15.173:6819/49288782","heartbeat_back_addr":"172.21.15.173:6823/49288782","heartbeat_front_addr":"172.21.15.173:6821/49288782","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:10:55.172572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:11:22.140314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:11:49.151682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:12:12.352626+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:12:37.457216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:13:02.197643+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/1955086590":"2024-02-11T06:09:42.089772+0000","172.21.15.89:0/644693661":"2024-02-11T06:09:42.089772+0000","172.21.15.89:0/1015341098":"2024-02-11T06:09:42.089772+0000","172.21.15.89:6800/1955086590":"2024-02-11T06:09:42.089772+0000","172.21.15.89:0/3545264344":"2024-02-11T06:08:38.557858+0000","172.21.15.89:6800/1397975713":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/1754666852":"2024-02-11T06:09:42.089772+0000","172.21.15.89:6800/3711070832":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/2946515553":"2024-02-11T06:08:08.869122+0000","172.21.15.89:6801/3711070832":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/2878753730":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/478488956":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/2858441641":"2024-02-11T06:08:08.869122+0000","172.21.15.89:6801/1397975713":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/632061317":"2024-02-11T06:08:38.557858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:13:23.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.237+0000 7f864236c700 1 -- 172.21.15.89:0/278553345 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f86240692a0 msgr2=0x7f862406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:23.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.237+0000 7f864236c700 1 --2- 172.21.15.89:0/278553345 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f86240692a0 0x7f862406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f862c0041d0 tx=0x7f862c0043c0 comp rx=0 tx=0).stop 2024-02-10T06:13:23.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.237+0000 7f864236c700 1 -- 172.21.15.89:0/278553345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863c102800 msgr2=0x7f863c114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:23.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.237+0000 7f864236c700 1 --2- 172.21.15.89:0/278553345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863c102800 0x7f863c114020 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f86300090d0 tx=0x7f8630009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:23.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.238+0000 7f864236c700 1 -- 172.21.15.89:0/278553345 shutdown_connections 2024-02-10T06:13:23.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.238+0000 7f864236c700 1 --2- 172.21.15.89:0/278553345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f863c101e60 0x7f863c113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:23.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.238+0000 7f864236c700 1 --2- 172.21.15.89:0/278553345 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f86240692a0 0x7f862406b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:23.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.238+0000 7f864236c700 1 --2- 172.21.15.89:0/278553345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f863c102800 0x7f863c114020 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:23.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.238+0000 7f864236c700 1 -- 172.21.15.89:0/278553345 >> 172.21.15.89:0/278553345 conn(0x7f863c0f7a10 msgr2=0x7f863c066050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:23.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.238+0000 7f864236c700 1 -- 172.21.15.89:0/278553345 shutdown_connections 2024-02-10T06:13:23.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:23.238+0000 7f864236c700 1 -- 172.21.15.89:0/278553345 wait complete. 2024-02-10T06:13:23.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:23 smithi089 ceph-mon[29217]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:23.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:23 smithi173 ceph-mon[36251]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:23.998 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T06:13:23.999 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T06:13:23.999 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T06:13:24.000 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T06:13:24.000 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T06:13:24.001 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T06:13:24.341 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:24.342 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:24.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:24 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/278553345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:13:24.405 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:24.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:24 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/278553345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:13:25.432 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:25 smithi089 ceph-mon[29217]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:25.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:25 smithi173 ceph-mon[36251]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:25.655 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:25.655 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:27.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:27 smithi089 ceph-mon[29217]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:27.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:27 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:27.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:27 smithi173 ceph-mon[36251]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:27.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:27 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:28.827 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:29.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:28 smithi089 ceph-mon[29217]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:29.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.958+0000 7f81dcb92700 1 -- 172.21.15.89:0/1646904490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81d8101e50 msgr2=0x7f81d8102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:29.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.958+0000 7f81dcb92700 1 --2- 172.21.15.89:0/1646904490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81d8101e50 0x7f81d8102220 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f81c8004940 tx=0x7f81c8009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.958+0000 7f81dcb92700 1 -- 172.21.15.89:0/1646904490 shutdown_connections 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.958+0000 7f81dcb92700 1 --2- 172.21.15.89:0/1646904490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81d81027f0 0x7f81d80fbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.958+0000 7f81dcb92700 1 --2- 172.21.15.89:0/1646904490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81d8101e50 0x7f81d8102220 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.958+0000 7f81dcb92700 1 -- 172.21.15.89:0/1646904490 >> 172.21.15.89:0/1646904490 conn(0x7f81d80f79e0 msgr2=0x7f81d80f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.958+0000 7f81dcb92700 1 -- 172.21.15.89:0/1646904490 shutdown_connections 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.958+0000 7f81dcb92700 1 -- 172.21.15.89:0/1646904490 wait complete. 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.959+0000 7f81dcb92700 1 Processor -- start 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.959+0000 7f81dcb92700 1 -- start start 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.959+0000 7f81dcb92700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81d8101e50 0x7f81d81129f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.959+0000 7f81dcb92700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81d81027f0 0x7f81d8112f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.959+0000 7f81dcb92700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d8113610 con 0x7f81d81027f0 2024-02-10T06:13:29.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.959+0000 7f81dcb92700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d810c8b0 con 0x7f81d8101e50 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81d5d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81d81027f0 0x7f81d8112f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81d5d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81d81027f0 0x7f81d8112f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54486/0 (socket says 172.21.15.89:54486) 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81d5d9b700 1 -- 172.21.15.89:0/4226307282 learned_addr learned my addr 172.21.15.89:0/4226307282 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81d659c700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81d8101e50 0x7f81d81129f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81d5d9b700 1 -- 172.21.15.89:0/4226307282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81d8101e50 msgr2=0x7f81d81129f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81d5d9b700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81d8101e50 0x7f81d81129f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81d5d9b700 1 -- 172.21.15.89:0/4226307282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c8005040 con 0x7f81d81027f0 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81d659c700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81d8101e50 0x7f81d81129f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81d5d9b700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81d81027f0 0x7f81d8112f30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00daf0 tx=0x7f81cc00b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81c6ffd700 1 -- 172.21.15.89:0/4226307282 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81cc00ed70 con 0x7f81d81027f0 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81dcb92700 1 -- 172.21.15.89:0/4226307282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d810cb30 con 0x7f81d81027f0 2024-02-10T06:13:29.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81c6ffd700 1 -- 172.21.15.89:0/4226307282 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81cc017940 con 0x7f81d81027f0 2024-02-10T06:13:29.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81c6ffd700 1 -- 172.21.15.89:0/4226307282 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81cc016ac0 con 0x7f81d81027f0 2024-02-10T06:13:29.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.960+0000 7f81dcb92700 1 -- 172.21.15.89:0/4226307282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d810d070 con 0x7f81d81027f0 2024-02-10T06:13:29.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.961+0000 7f81dcb92700 1 -- 172.21.15.89:0/4226307282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f81d810d570 con 0x7f81d81027f0 2024-02-10T06:13:29.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.962+0000 7f81c6ffd700 1 -- 172.21.15.89:0/4226307282 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f81cc014070 con 0x7f81d81027f0 2024-02-10T06:13:29.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.962+0000 7f81c6ffd700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f81c00692a0 0x7f81c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:29.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.962+0000 7f81c6ffd700 1 -- 172.21.15.89:0/4226307282 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f81cc089040 con 0x7f81d81027f0 2024-02-10T06:13:29.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.962+0000 7f81d659c700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f81c00692a0 0x7f81c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:29.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.962+0000 7f81c6ffd700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236] conn(0x7f81c006ef70 0x7f81c0071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:29.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.962+0000 7f81c6ffd700 1 -- 172.21.15.89:0/4226307282 --> [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81c0071a30 con 0x7f81c006ef70 2024-02-10T06:13:29.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.962+0000 7f81c6ffd700 1 -- 172.21.15.89:0/4226307282 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81cc08b690 con 0x7f81d81027f0 2024-02-10T06:13:29.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.963+0000 7f81d659c700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f81c00692a0 0x7f81c006b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f81c8004f90 tx=0x7f81c8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:29.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.963+0000 7f81d6d9d700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236] conn(0x7f81c006ef70 0x7f81c0071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:29.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.963+0000 7f81d6d9d700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236] conn(0x7f81c006ef70 0x7f81c0071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:29.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.964+0000 7f81c6ffd700 1 -- 172.21.15.89:0/4226307282 <== osd.3 v2:172.21.15.173:6800/2644357236 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f81c0071a30 con 0x7f81c006ef70 2024-02-10T06:13:29.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.979+0000 7f81dcb92700 1 -- 172.21.15.89:0/4226307282 --> [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f81d8066210 con 0x7f81c006ef70 2024-02-10T06:13:29.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81c6ffd700 1 -- 172.21.15.89:0/4226307282 <== osd.3 v2:172.21.15.173:6800/2644357236 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f81d8066210 con 0x7f81c006ef70 2024-02-10T06:13:29.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 -- 172.21.15.89:0/4226307282 >> [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236] conn(0x7f81c006ef70 msgr2=0x7f81c0071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:29.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236] conn(0x7f81c006ef70 0x7f81c0071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:29.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 -- 172.21.15.89:0/4226307282 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f81c00692a0 msgr2=0x7f81c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:29.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f81c00692a0 0x7f81c006b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f81c8004f90 tx=0x7f81c8009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:29.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 -- 172.21.15.89:0/4226307282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81d81027f0 msgr2=0x7f81d8112f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:29.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81d81027f0 0x7f81d8112f30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00daf0 tx=0x7f81cc00b4c0 comp rx=0 tx=0).stop 2024-02-10T06:13:29.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 -- 172.21.15.89:0/4226307282 shutdown_connections 2024-02-10T06:13:29.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.173:6800/2644357236,v1:172.21.15.173:6801/2644357236] conn(0x7f81c006ef70 0x7f81c0071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:29.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81d8101e50 0x7f81d81129f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:29.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f81c00692a0 0x7f81c006b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:29.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 --2- 172.21.15.89:0/4226307282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81d81027f0 0x7f81d8112f30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:29.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 -- 172.21.15.89:0/4226307282 >> 172.21.15.89:0/4226307282 conn(0x7f81d80f79e0 msgr2=0x7f81d80f9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:29.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 -- 172.21.15.89:0/4226307282 shutdown_connections 2024-02-10T06:13:29.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:27.980+0000 7f81dcb92700 1 -- 172.21.15.89:0/4226307282 wait complete. 2024-02-10T06:13:29.399 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:28 smithi173 ceph-mon[36251]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:29.979 INFO:teuthology.orchestra.run.smithi089.stdout:115964117009 2024-02-10T06:13:29.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T06:13:30.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.400+0000 7f04ae544700 1 -- 172.21.15.89:0/3909767897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04a80fe8f0 msgr2=0x7f04a8107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:30.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.400+0000 7f04ae544700 1 --2- 172.21.15.89:0/3909767897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04a80fe8f0 0x7f04a8107050 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f049c004970 tx=0x7f049c009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:30.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.400+0000 7f04ae544700 1 -- 172.21.15.89:0/3909767897 shutdown_connections 2024-02-10T06:13:30.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.400+0000 7f04ae544700 1 --2- 172.21.15.89:0/3909767897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04a80fe8f0 0x7f04a8107050 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:30.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.400+0000 7f04ae544700 1 --2- 172.21.15.89:0/3909767897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a80fdfe0 0x7f04a80fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:30.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.400+0000 7f04ae544700 1 -- 172.21.15.89:0/3909767897 >> 172.21.15.89:0/3909767897 conn(0x7f04a8071d90 msgr2=0x7f04a8072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.400+0000 7f04ae544700 1 -- 172.21.15.89:0/3909767897 shutdown_connections 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.401+0000 7f04ae544700 1 -- 172.21.15.89:0/3909767897 wait complete. 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.401+0000 7f04ae544700 1 Processor -- start 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.401+0000 7f04ae544700 1 -- start start 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.401+0000 7f04ae544700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04a80fdfe0 0x7f04a81139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.401+0000 7f04ae544700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a80fe8f0 0x7f04a8113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.401+0000 7f04ae544700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a8114580 con 0x7f04a80fdfe0 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.401+0000 7f04ae544700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a810d8b0 con 0x7f04a80fe8f0 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.401+0000 7f04a7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04a80fdfe0 0x7f04a81139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.401+0000 7f04a7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04a80fdfe0 0x7f04a81139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46982/0 (socket says 172.21.15.89:46982) 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.401+0000 7f04a7fff700 1 -- 172.21.15.89:0/4170430370 learned_addr learned my addr 172.21.15.89:0/4170430370 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.402+0000 7f04a77fe700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a80fe8f0 0x7f04a8113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.402+0000 7f04a7fff700 1 -- 172.21.15.89:0/4170430370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a80fe8f0 msgr2=0x7f04a8113f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:30.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.402+0000 7f04a7fff700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a80fe8f0 0x7f04a8113f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.402+0000 7f04a7fff700 1 -- 172.21.15.89:0/4170430370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c005040 con 0x7f04a80fdfe0 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.402+0000 7f04a7fff700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04a80fdfe0 0x7f04a81139f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f049800c830 tx=0x7f0498004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.402+0000 7f04a4ff9700 1 -- 172.21.15.89:0/4170430370 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f049800a760 con 0x7f04a80fdfe0 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.402+0000 7f04ae544700 1 -- 172.21.15.89:0/4170430370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a810db90 con 0x7f04a80fdfe0 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.402+0000 7f04a4ff9700 1 -- 172.21.15.89:0/4170430370 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f049800ad90 con 0x7f04a80fdfe0 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.402+0000 7f04a4ff9700 1 -- 172.21.15.89:0/4170430370 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04980113a0 con 0x7f04a80fdfe0 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.402+0000 7f04ae544700 1 -- 172.21.15.89:0/4170430370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a810e0a0 con 0x7f04a80fdfe0 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.403+0000 7f04ae544700 1 -- 172.21.15.89:0/4170430370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f04a810e5a0 con 0x7f04a80fdfe0 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.403+0000 7f04a4ff9700 1 -- 172.21.15.89:0/4170430370 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0498012400 con 0x7f04a80fdfe0 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.404+0000 7f04a4ff9700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f04900692a0 0x7f049006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.404+0000 7f04a4ff9700 1 -- 172.21.15.89:0/4170430370 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0498086ef0 con 0x7f04a80fdfe0 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.404+0000 7f04a77fe700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f04900692a0 0x7f049006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:30.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.404+0000 7f04a4ff9700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828] conn(0x7f049006ef70 0x7f0490071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:30.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.404+0000 7f04a4ff9700 1 -- 172.21.15.89:0/4170430370 --> [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0490071a30 con 0x7f049006ef70 2024-02-10T06:13:30.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.404+0000 7f04a4ff9700 1 -- 172.21.15.89:0/4170430370 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f049808c4f0 con 0x7f04a80fdfe0 2024-02-10T06:13:30.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.404+0000 7f04acae1700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828] conn(0x7f049006ef70 0x7f0490071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:30.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.404+0000 7f04a77fe700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f04900692a0 0x7f049006b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f04a810ebe0 tx=0x7f049c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:30.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.404+0000 7f04acae1700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828] conn(0x7f049006ef70 0x7f0490071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:30.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.405+0000 7f04a4ff9700 1 -- 172.21.15.89:0/4170430370 <== osd.1 v2:172.21.15.89:6810/3051839828 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f0490071a30 con 0x7f049006ef70 2024-02-10T06:13:30.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 -- 172.21.15.89:0/4170430370 --> [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f04a80738b0 con 0x7f049006ef70 2024-02-10T06:13:30.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04a4ff9700 1 -- 172.21.15.89:0/4170430370 <== osd.1 v2:172.21.15.89:6810/3051839828 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f04a80738b0 con 0x7f049006ef70 2024-02-10T06:13:30.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 -- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828] conn(0x7f049006ef70 msgr2=0x7f0490071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:30.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828] conn(0x7f049006ef70 0x7f0490071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:30.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 -- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f04900692a0 msgr2=0x7f049006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:30.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f04900692a0 0x7f049006b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f04a810ebe0 tx=0x7f049c009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:30.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 -- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04a80fdfe0 msgr2=0x7f04a81139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:30.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04a80fdfe0 0x7f04a81139f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f049800c830 tx=0x7f0498004470 comp rx=0 tx=0).stop 2024-02-10T06:13:30.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 -- 172.21.15.89:0/4170430370 shutdown_connections 2024-02-10T06:13:30.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6810/3051839828,v1:172.21.15.89:6811/3051839828] conn(0x7f049006ef70 0x7f0490071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:30.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f04900692a0 0x7f049006b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:30.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04a80fdfe0 0x7f04a81139f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:30.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 --2- 172.21.15.89:0/4170430370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a80fe8f0 0x7f04a8113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:30.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.420+0000 7f04ae544700 1 -- 172.21.15.89:0/4170430370 >> 172.21.15.89:0/4170430370 conn(0x7f04a8071d90 msgr2=0x7f04a8101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:30.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.421+0000 7f04ae544700 1 -- 172.21.15.89:0/4170430370 shutdown_connections 2024-02-10T06:13:30.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:30.421+0000 7f04ae544700 1 -- 172.21.15.89:0/4170430370 wait complete. 2024-02-10T06:13:31.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:31 smithi089 ceph-mon[29217]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:31.543 INFO:teuthology.orchestra.run.smithi089.stdout:60129542173 2024-02-10T06:13:31.543 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T06:13:31.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:31 smithi173 ceph-mon[36251]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:32.797 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:33.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.061+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3018395014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a0101e40 msgr2=0x7f56a0102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:33.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.061+0000 7f56a6a8f700 1 --2- 172.21.15.89:0/3018395014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a0101e40 0x7f56a0102210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5694004940 tx=0x7f5694009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:33.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.061+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3018395014 shutdown_connections 2024-02-10T06:13:33.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.061+0000 7f56a6a8f700 1 --2- 172.21.15.89:0/3018395014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56a01027e0 0x7f56a00fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.061+0000 7f56a6a8f700 1 --2- 172.21.15.89:0/3018395014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a0101e40 0x7f56a0102210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.061+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3018395014 >> 172.21.15.89:0/3018395014 conn(0x7f56a0071d90 msgr2=0x7f56a0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:33.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.061+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3018395014 shutdown_connections 2024-02-10T06:13:33.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.061+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3018395014 wait complete. 2024-02-10T06:13:33.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a6a8f700 1 Processor -- start 2024-02-10T06:13:33.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a6a8f700 1 -- start start 2024-02-10T06:13:33.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a6a8f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a0101e40 0x7f56a0102ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:33.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a6a8f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56a01027e0 0x7f56a01031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:33.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a6a8f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a0103870 con 0x7f56a0101e40 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a6a8f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a01039e0 con 0x7f56a01027e0 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a482b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a0101e40 0x7f56a0102ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a482b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a0101e40 0x7f56a0102ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47008/0 (socket says 172.21.15.89:47008) 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a482b700 1 -- 172.21.15.89:0/3862326359 learned_addr learned my addr 172.21.15.89:0/3862326359 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f569ffff700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56a01027e0 0x7f56a01031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a482b700 1 -- 172.21.15.89:0/3862326359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56a01027e0 msgr2=0x7f56a01031e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a482b700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56a01027e0 0x7f56a01031e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.062+0000 7f56a482b700 1 -- 172.21.15.89:0/3862326359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5694005040 con 0x7f56a0101e40 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.063+0000 7f56a482b700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a0101e40 0x7f56a0102ca0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f569400b3b0 tx=0x7f5694009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.063+0000 7f569d7fa700 1 -- 172.21.15.89:0/3862326359 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f569400e040 con 0x7f56a0101e40 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.063+0000 7f569d7fa700 1 -- 172.21.15.89:0/3862326359 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f569400bbf0 con 0x7f56a0101e40 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.063+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3862326359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a0119cc0 con 0x7f56a0101e40 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.063+0000 7f569d7fa700 1 -- 172.21.15.89:0/3862326359 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f569401c630 con 0x7f56a0101e40 2024-02-10T06:13:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.063+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3862326359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a011a1b0 con 0x7f56a0101e40 2024-02-10T06:13:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.064+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3862326359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f56a00660d0 con 0x7f56a0101e40 2024-02-10T06:13:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.064+0000 7f569d7fa700 1 -- 172.21.15.89:0/3862326359 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f569400bd60 con 0x7f56a0101e40 2024-02-10T06:13:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.065+0000 7f569d7fa700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f56900692a0 0x7f569006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.065+0000 7f569d7fa700 1 -- 172.21.15.89:0/3862326359 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5694088240 con 0x7f56a0101e40 2024-02-10T06:13:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.065+0000 7f569ffff700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f56900692a0 0x7f569006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.065+0000 7f569d7fa700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782] conn(0x7f569006ef70 0x7f5690071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.065+0000 7f569d7fa700 1 -- 172.21.15.89:0/3862326359 --> [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5690071a30 con 0x7f569006ef70 2024-02-10T06:13:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.065+0000 7f569d7fa700 1 -- 172.21.15.89:0/3862326359 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f569408a830 con 0x7f56a0101e40 2024-02-10T06:13:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.065+0000 7f569ffff700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f56900692a0 0x7f569006b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f568c006fd0 tx=0x7f568c008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.065+0000 7f56a502c700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782] conn(0x7f569006ef70 0x7f5690071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.066+0000 7f56a502c700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782] conn(0x7f569006ef70 0x7f5690071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.066+0000 7f569d7fa700 1 -- 172.21.15.89:0/3862326359 <== osd.5 v2:172.21.15.173:6816/49288782 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5690071a30 con 0x7f569006ef70 2024-02-10T06:13:33.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.081+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3862326359 --> [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f56a00738b0 con 0x7f569006ef70 2024-02-10T06:13:33.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f569d7fa700 1 -- 172.21.15.89:0/3862326359 <== osd.5 v2:172.21.15.173:6816/49288782 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f56a00738b0 con 0x7f569006ef70 2024-02-10T06:13:33.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3862326359 >> [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782] conn(0x7f569006ef70 msgr2=0x7f5690071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:33.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782] conn(0x7f569006ef70 0x7f5690071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3862326359 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f56900692a0 msgr2=0x7f569006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:33.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f56900692a0 0x7f569006b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f568c006fd0 tx=0x7f568c008040 comp rx=0 tx=0).stop 2024-02-10T06:13:33.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3862326359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a0101e40 msgr2=0x7f56a0102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:33.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a0101e40 0x7f56a0102ca0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f569400b3b0 tx=0x7f5694009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:33.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3862326359 shutdown_connections 2024-02-10T06:13:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f56900692a0 0x7f569006b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a0101e40 0x7f56a0102ca0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.173:6816/49288782,v1:172.21.15.173:6817/49288782] conn(0x7f569006ef70 0x7f5690071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 --2- 172.21.15.89:0/3862326359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56a01027e0 0x7f56a01031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3862326359 >> 172.21.15.89:0/3862326359 conn(0x7f56a0071d90 msgr2=0x7f56a00656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3862326359 shutdown_connections 2024-02-10T06:13:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.082+0000 7f56a6a8f700 1 -- 172.21.15.89:0/3862326359 wait complete. 2024-02-10T06:13:33.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:33 smithi089 ceph-mon[29217]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:33.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:33 smithi173 ceph-mon[36251]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:33.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.745+0000 7f23f678c700 1 -- 172.21.15.89:0/127040288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23f01027f0 msgr2=0x7f23f00fbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:33.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.745+0000 7f23f678c700 1 --2- 172.21.15.89:0/127040288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23f01027f0 0x7f23f00fbe60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f23e4004940 tx=0x7f23e4009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:33.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.746+0000 7f23f678c700 1 -- 172.21.15.89:0/127040288 shutdown_connections 2024-02-10T06:13:33.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.746+0000 7f23f678c700 1 --2- 172.21.15.89:0/127040288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23f01027f0 0x7f23f00fbe60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.746+0000 7f23f678c700 1 --2- 172.21.15.89:0/127040288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f0101e50 0x7f23f0102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.746+0000 7f23f678c700 1 -- 172.21.15.89:0/127040288 >> 172.21.15.89:0/127040288 conn(0x7f23f00f79c0 msgr2=0x7f23f00f9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:33.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.746+0000 7f23f678c700 1 -- 172.21.15.89:0/127040288 shutdown_connections 2024-02-10T06:13:33.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.746+0000 7f23f678c700 1 -- 172.21.15.89:0/127040288 wait complete. 2024-02-10T06:13:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.747+0000 7f23f678c700 1 Processor -- start 2024-02-10T06:13:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.747+0000 7f23f678c700 1 -- start start 2024-02-10T06:13:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.747+0000 7f23f678c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23f0101e50 0x7f23f0110820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.747+0000 7f23f678c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f01027f0 0x7f23f0110da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.747+0000 7f23f678c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f01113f0 con 0x7f23f0101e50 2024-02-10T06:13:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.747+0000 7f23f678c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f010b760 con 0x7f23f01027f0 2024-02-10T06:13:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.747+0000 7f23effff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23f0101e50 0x7f23f0110820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:33.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.747+0000 7f23effff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23f0101e50 0x7f23f0110820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47018/0 (socket says 172.21.15.89:47018) 2024-02-10T06:13:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.747+0000 7f23effff700 1 -- 172.21.15.89:0/4274318149 learned_addr learned my addr 172.21.15.89:0/4274318149 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.748+0000 7f23ef7fe700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f01027f0 0x7f23f0110da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.748+0000 7f23effff700 1 -- 172.21.15.89:0/4274318149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f01027f0 msgr2=0x7f23f0110da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.748+0000 7f23effff700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f01027f0 0x7f23f0110da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.748+0000 7f23effff700 1 -- 172.21.15.89:0/4274318149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e4005040 con 0x7f23f0101e50 2024-02-10T06:13:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.748+0000 7f23effff700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23f0101e50 0x7f23f0110820 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f23e0002a50 tx=0x7f23e00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.748+0000 7f23ecff9700 1 -- 172.21.15.89:0/4274318149 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e0007cc0 con 0x7f23f0101e50 2024-02-10T06:13:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.748+0000 7f23f678c700 1 -- 172.21.15.89:0/4274318149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f010ba10 con 0x7f23f0101e50 2024-02-10T06:13:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.748+0000 7f23ecff9700 1 -- 172.21.15.89:0/4274318149 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23e000f040 con 0x7f23f0101e50 2024-02-10T06:13:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.748+0000 7f23ecff9700 1 -- 172.21.15.89:0/4274318149 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e000da30 con 0x7f23f0101e50 2024-02-10T06:13:33.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.748+0000 7f23f678c700 1 -- 172.21.15.89:0/4274318149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f010bf20 con 0x7f23f0101e50 2024-02-10T06:13:33.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.749+0000 7f23f678c700 1 -- 172.21.15.89:0/4274318149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f23f010c380 con 0x7f23f0101e50 2024-02-10T06:13:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.750+0000 7f23ecff9700 1 -- 172.21.15.89:0/4274318149 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f23e000db90 con 0x7f23f0101e50 2024-02-10T06:13:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.750+0000 7f23ecff9700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f23d80692a0 0x7f23d806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.750+0000 7f23ecff9700 1 -- 172.21.15.89:0/4274318149 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f23e0086c20 con 0x7f23f0101e50 2024-02-10T06:13:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.750+0000 7f23ef7fe700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f23d80692a0 0x7f23d806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.750+0000 7f23ecff9700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040] conn(0x7f23d806ef70 0x7f23d8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.750+0000 7f23ecff9700 1 -- 172.21.15.89:0/4274318149 --> [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23d8071a30 con 0x7f23d806ef70 2024-02-10T06:13:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.750+0000 7f23ecff9700 1 -- 172.21.15.89:0/4274318149 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23e008c280 con 0x7f23f0101e50 2024-02-10T06:13:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.751+0000 7f23f4d29700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040] conn(0x7f23d806ef70 0x7f23d8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.751+0000 7f23ef7fe700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f23d80692a0 0x7f23d806b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f23e400b3b0 tx=0x7f23e4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:33.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.751+0000 7f23f4d29700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040] conn(0x7f23d806ef70 0x7f23d8071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:33.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.752+0000 7f23ecff9700 1 -- 172.21.15.89:0/4274318149 <== osd.0 v2:172.21.15.89:6802/2704721040 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f23d8071a30 con 0x7f23d806ef70 2024-02-10T06:13:33.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.766+0000 7f23f678c700 1 -- 172.21.15.89:0/4274318149 --> [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f23f00fb5e0 con 0x7f23d806ef70 2024-02-10T06:13:33.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.766+0000 7f23ecff9700 1 -- 172.21.15.89:0/4274318149 <== osd.0 v2:172.21.15.89:6802/2704721040 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f23f00fb5e0 con 0x7f23d806ef70 2024-02-10T06:13:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 -- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040] conn(0x7f23d806ef70 msgr2=0x7f23d8071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040] conn(0x7f23d806ef70 0x7f23d8071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 -- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f23d80692a0 msgr2=0x7f23d806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f23d80692a0 0x7f23d806b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f23e400b3b0 tx=0x7f23e4009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 -- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23f0101e50 msgr2=0x7f23f0110820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23f0101e50 0x7f23f0110820 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f23e0002a50 tx=0x7f23e00046b0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 -- 172.21.15.89:0/4274318149 shutdown_connections 2024-02-10T06:13:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6802/2704721040,v1:172.21.15.89:6803/2704721040] conn(0x7f23d806ef70 0x7f23d8071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f23d80692a0 0x7f23d806b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23f0101e50 0x7f23f0110820 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 --2- 172.21.15.89:0/4274318149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23f01027f0 0x7f23f0110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 -- 172.21.15.89:0/4274318149 >> 172.21.15.89:0/4274318149 conn(0x7f23f00f79c0 msgr2=0x7f23f0066150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 -- 172.21.15.89:0/4274318149 shutdown_connections 2024-02-10T06:13:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:33.767+0000 7f23f678c700 1 -- 172.21.15.89:0/4274318149 wait complete. 2024-02-10T06:13:34.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.513+0000 7f611942a700 1 -- 172.21.15.89:0/1004410189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6114102800 msgr2=0x7f61140fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.513+0000 7f611942a700 1 --2- 172.21.15.89:0/1004410189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6114102800 0x7f61140fbe70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6108004940 tx=0x7f6108009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.513+0000 7f611942a700 1 -- 172.21.15.89:0/1004410189 shutdown_connections 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.513+0000 7f611942a700 1 --2- 172.21.15.89:0/1004410189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6114102800 0x7f61140fbe70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.513+0000 7f611942a700 1 --2- 172.21.15.89:0/1004410189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6114101e60 0x7f6114102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.513+0000 7f611942a700 1 -- 172.21.15.89:0/1004410189 >> 172.21.15.89:0/1004410189 conn(0x7f61140f7a10 msgr2=0x7f61140f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.513+0000 7f611942a700 1 -- 172.21.15.89:0/1004410189 shutdown_connections 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.513+0000 7f611942a700 1 -- 172.21.15.89:0/1004410189 wait complete. 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f611942a700 1 Processor -- start 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f611942a700 1 -- start start 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f611942a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6114101e60 0x7f611410eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f611942a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6114102800 0x7f611410f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f611942a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611410f6e0 con 0x7f6114102800 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f611942a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611410f820 con 0x7f6114101e60 2024-02-10T06:13:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f61127fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6114102800 0x7f611410f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:34.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f61127fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6114102800 0x7f611410f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47038/0 (socket says 172.21.15.89:47038) 2024-02-10T06:13:34.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f61127fc700 1 -- 172.21.15.89:0/2022258060 learned_addr learned my addr 172.21.15.89:0/2022258060 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:34.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f6112ffd700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6114101e60 0x7f611410eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:34.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f61127fc700 1 -- 172.21.15.89:0/2022258060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6114101e60 msgr2=0x7f611410eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:34.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f61127fc700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6114101e60 0x7f611410eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:34.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.514+0000 7f61127fc700 1 -- 172.21.15.89:0/2022258060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6108005040 con 0x7f6114102800 2024-02-10T06:13:34.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.515+0000 7f61127fc700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6114102800 0x7f611410f0c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6108009100 tx=0x7f6108009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.515+0000 7f60fb7fe700 1 -- 172.21.15.89:0/2022258060 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f610800e040 con 0x7f6114102800 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.515+0000 7f611942a700 1 -- 172.21.15.89:0/2022258060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611410c7b0 con 0x7f6114102800 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.515+0000 7f60fb7fe700 1 -- 172.21.15.89:0/2022258060 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f610800be00 con 0x7f6114102800 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.515+0000 7f60fb7fe700 1 -- 172.21.15.89:0/2022258060 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6108013860 con 0x7f6114102800 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.515+0000 7f611942a700 1 -- 172.21.15.89:0/2022258060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611410ccf0 con 0x7f6114102800 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.515+0000 7f611942a700 1 -- 172.21.15.89:0/2022258060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f611405e6d0 con 0x7f6114102800 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.516+0000 7f60fb7fe700 1 -- 172.21.15.89:0/2022258060 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f610801b580 con 0x7f6114102800 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.517+0000 7f60fb7fe700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f60fc0692a0 0x7f60fc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.517+0000 7f60fb7fe700 1 -- 172.21.15.89:0/2022258060 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f610808be60 con 0x7f6114102800 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.517+0000 7f6112ffd700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f60fc0692a0 0x7f60fc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.517+0000 7f60fb7fe700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780] conn(0x7f60fc06ef70 0x7f60fc071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.517+0000 7f60fb7fe700 1 -- 172.21.15.89:0/2022258060 --> [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60fc071a30 con 0x7f60fc06ef70 2024-02-10T06:13:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.517+0000 7f60fb7fe700 1 -- 172.21.15.89:0/2022258060 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6108091460 con 0x7f6114102800 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.517+0000 7f61137fe700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780] conn(0x7f60fc06ef70 0x7f60fc071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.517+0000 7f6112ffd700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f60fc0692a0 0x7f60fc06b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f610c007540 tx=0x7f610c009780 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.518+0000 7f61137fe700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780] conn(0x7f60fc06ef70 0x7f60fc071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.518+0000 7f60fb7fe700 1 -- 172.21.15.89:0/2022258060 <== osd.4 v2:172.21.15.173:6808/2725914780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f60fc071a30 con 0x7f60fc06ef70 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.533+0000 7f611942a700 1 -- 172.21.15.89:0/2022258060 --> [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f611405eb50 con 0x7f60fc06ef70 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f60fb7fe700 1 -- 172.21.15.89:0/2022258060 <== osd.4 v2:172.21.15.173:6808/2725914780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f611405eb50 con 0x7f60fc06ef70 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 -- 172.21.15.89:0/2022258060 >> [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780] conn(0x7f60fc06ef70 msgr2=0x7f60fc071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780] conn(0x7f60fc06ef70 0x7f60fc071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 -- 172.21.15.89:0/2022258060 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f60fc0692a0 msgr2=0x7f60fc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f60fc0692a0 0x7f60fc06b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f610c007540 tx=0x7f610c009780 comp rx=0 tx=0).stop 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 -- 172.21.15.89:0/2022258060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6114102800 msgr2=0x7f611410f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6114102800 0x7f611410f0c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6108009100 tx=0x7f6108009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 -- 172.21.15.89:0/2022258060 shutdown_connections 2024-02-10T06:13:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.173:6808/2725914780,v1:172.21.15.173:6809/2725914780] conn(0x7f60fc06ef70 0x7f60fc071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:34.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6114101e60 0x7f611410eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:34.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f60fc0692a0 0x7f60fc06b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:34.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 --2- 172.21.15.89:0/2022258060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6114102800 0x7f611410f0c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:34.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 -- 172.21.15.89:0/2022258060 >> 172.21.15.89:0/2022258060 conn(0x7f61140f7a10 msgr2=0x7f61140f9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:34.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 -- 172.21.15.89:0/2022258060 shutdown_connections 2024-02-10T06:13:34.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:34.534+0000 7f611942a700 1 -- 172.21.15.89:0/2022258060 wait complete. 2024-02-10T06:13:35.162 INFO:teuthology.orchestra.run.smithi089.stdout:150323855374 2024-02-10T06:13:35.162 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T06:13:35.541 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:35 smithi089 ceph-mon[29217]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:35.564 INFO:teuthology.orchestra.run.smithi089.stdout:171798691849 2024-02-10T06:13:35.564 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T06:13:35.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:35 smithi173 ceph-mon[36251]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:36.213 INFO:teuthology.orchestra.run.smithi089.stdout:38654705698 2024-02-10T06:13:36.213 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T06:13:37.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:37 smithi089 ceph-mon[29217]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:37.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.392+0000 7fafe6d9c700 1 -- 172.21.15.89:0/3157651623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00fe8f0 msgr2=0x7fafe0107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:37.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.392+0000 7fafe6d9c700 1 --2- 172.21.15.89:0/3157651623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00fe8f0 0x7fafe0107050 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fafcc004970 tx=0x7fafcc009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.392+0000 7fafe6d9c700 1 -- 172.21.15.89:0/3157651623 shutdown_connections 2024-02-10T06:13:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.392+0000 7fafe6d9c700 1 --2- 172.21.15.89:0/3157651623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00fe8f0 0x7fafe0107050 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.392+0000 7fafe6d9c700 1 --2- 172.21.15.89:0/3157651623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafe00fdfe0 0x7fafe00fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.392+0000 7fafe6d9c700 1 -- 172.21.15.89:0/3157651623 >> 172.21.15.89:0/3157651623 conn(0x7fafe0071d90 msgr2=0x7fafe0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.392+0000 7fafe6d9c700 1 -- 172.21.15.89:0/3157651623 shutdown_connections 2024-02-10T06:13:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.392+0000 7fafe6d9c700 1 -- 172.21.15.89:0/3157651623 wait complete. 2024-02-10T06:13:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.393+0000 7fafe6d9c700 1 Processor -- start 2024-02-10T06:13:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.393+0000 7fafe6d9c700 1 -- start start 2024-02-10T06:13:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.393+0000 7fafe6d9c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafe00fdfe0 0x7fafe0111860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.393+0000 7fafe6d9c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00fe8f0 0x7fafe0111da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.393+0000 7fafe6d9c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe0112480 con 0x7fafe00fe8f0 2024-02-10T06:13:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.393+0000 7fafe6d9c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe010b720 con 0x7fafe00fdfe0 2024-02-10T06:13:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.393+0000 7fafdffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00fe8f0 0x7fafe0111da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.393+0000 7fafdffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00fe8f0 0x7fafe0111da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47068/0 (socket says 172.21.15.89:47068) 2024-02-10T06:13:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.393+0000 7fafdffff700 1 -- 172.21.15.89:0/2000900742 learned_addr learned my addr 172.21.15.89:0/2000900742 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.394+0000 7fafe4b38700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafe00fdfe0 0x7fafe0111860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.394+0000 7fafdffff700 1 -- 172.21.15.89:0/2000900742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafe00fdfe0 msgr2=0x7fafe0111860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:37.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.394+0000 7fafdffff700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafe00fdfe0 0x7fafe0111860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:37.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.394+0000 7fafdffff700 1 -- 172.21.15.89:0/2000900742 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc005040 con 0x7fafe00fe8f0 2024-02-10T06:13:37.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.394+0000 7fafdffff700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00fe8f0 0x7fafe0111da0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fafcc000c00 tx=0x7fafcc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:37.397 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:37.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.394+0000 7fafdd7fa700 1 -- 172.21.15.89:0/2000900742 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafcc00e040 con 0x7fafe00fe8f0 2024-02-10T06:13:37.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.394+0000 7fafe6d9c700 1 -- 172.21.15.89:0/2000900742 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe010b9a0 con 0x7fafe00fe8f0 2024-02-10T06:13:37.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.394+0000 7fafdd7fa700 1 -- 172.21.15.89:0/2000900742 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fafcc009af0 con 0x7fafe00fe8f0 2024-02-10T06:13:37.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.394+0000 7fafdd7fa700 1 -- 172.21.15.89:0/2000900742 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafcc01c3f0 con 0x7fafe00fe8f0 2024-02-10T06:13:37.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.394+0000 7fafe6d9c700 1 -- 172.21.15.89:0/2000900742 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe010bee0 con 0x7fafe00fe8f0 2024-02-10T06:13:37.398 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.395+0000 7fafe6d9c700 1 -- 172.21.15.89:0/2000900742 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fafe010c440 con 0x7fafe00fe8f0 2024-02-10T06:13:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.396+0000 7fafdd7fa700 1 -- 172.21.15.89:0/2000900742 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fafcc013420 con 0x7fafe00fe8f0 2024-02-10T06:13:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.396+0000 7fafdd7fa700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fafd006d6b0 0x7fafd006fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.396+0000 7fafdd7fa700 1 -- 172.21.15.89:0/2000900742 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fafcc0880a0 con 0x7fafe00fe8f0 2024-02-10T06:13:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.396+0000 7fafe4b38700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fafd006d6b0 0x7fafd006fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.396+0000 7fafdd7fa700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379] conn(0x7fafd0073380 0x7fafd0075790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.396+0000 7fafdd7fa700 1 -- 172.21.15.89:0/2000900742 --> [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafd0075e40 con 0x7fafd0073380 2024-02-10T06:13:37.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.396+0000 7fafdd7fa700 1 -- 172.21.15.89:0/2000900742 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafcc08a690 con 0x7fafe00fe8f0 2024-02-10T06:13:37.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.397+0000 7fafe5339700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379] conn(0x7fafd0073380 0x7fafd0075790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:37.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.397+0000 7fafe4b38700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fafd006d6b0 0x7fafd006fb60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009bf0 tx=0x7fafd40091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:37.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.397+0000 7fafe5339700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379] conn(0x7fafd0073380 0x7fafd0075790 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:37.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.398+0000 7fafdd7fa700 1 -- 172.21.15.89:0/2000900742 <== osd.2 v2:172.21.15.89:6818/637372379 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fafd0075e40 con 0x7fafd0073380 2024-02-10T06:13:37.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.416+0000 7fafe6d9c700 1 -- 172.21.15.89:0/2000900742 --> [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fafe005e690 con 0x7fafd0073380 2024-02-10T06:13:37.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.417+0000 7fafdd7fa700 1 -- 172.21.15.89:0/2000900742 <== osd.2 v2:172.21.15.89:6818/637372379 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fafe005e690 con 0x7fafd0073380 2024-02-10T06:13:37.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.417+0000 7fafe6d9c700 1 -- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379] conn(0x7fafd0073380 msgr2=0x7fafd0075790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.417+0000 7fafe6d9c700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379] conn(0x7fafd0073380 0x7fafd0075790 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.417+0000 7fafe6d9c700 1 -- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fafd006d6b0 msgr2=0x7fafd006fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.417+0000 7fafe6d9c700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fafd006d6b0 0x7fafd006fb60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009bf0 tx=0x7fafd40091a0 comp rx=0 tx=0).stop 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.417+0000 7fafe6d9c700 1 -- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00fe8f0 msgr2=0x7fafe0111da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.417+0000 7fafe6d9c700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00fe8f0 0x7fafe0111da0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fafcc000c00 tx=0x7fafcc009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.418+0000 7fafe6d9c700 1 -- 172.21.15.89:0/2000900742 shutdown_connections 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.418+0000 7fafe6d9c700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafe00fdfe0 0x7fafe0111860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.418+0000 7fafe6d9c700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6818/637372379,v1:172.21.15.89:6819/637372379] conn(0x7fafd0073380 0x7fafd0075790 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.418+0000 7fafe6d9c700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fafd006d6b0 0x7fafd006fb60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.418+0000 7fafe6d9c700 1 --2- 172.21.15.89:0/2000900742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafe00fe8f0 0x7fafe0111da0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.418+0000 7fafe6d9c700 1 -- 172.21.15.89:0/2000900742 >> 172.21.15.89:0/2000900742 conn(0x7fafe0071d90 msgr2=0x7fafe0101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:37.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.418+0000 7fafe6d9c700 1 -- 172.21.15.89:0/2000900742 shutdown_connections 2024-02-10T06:13:37.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:37.418+0000 7fafe6d9c700 1 -- 172.21.15.89:0/2000900742 wait complete. 2024-02-10T06:13:37.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:37 smithi173 ceph-mon[36251]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:39.057 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:38 smithi089 ceph-mon[29217]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:39.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:38 smithi173 ceph-mon[36251]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:39.458 INFO:teuthology.orchestra.run.smithi089.stdout:81604378649 2024-02-10T06:13:39.459 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T06:13:40.227 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:40.259 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:41.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:41 smithi089 ceph-mon[29217]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:41.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:41 smithi173 ceph-mon[36251]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:42.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.052+0000 7f43a9b51700 1 -- 172.21.15.89:0/767253935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a4101e60 msgr2=0x7f43a4102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:42.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.052+0000 7f43a9b51700 1 --2- 172.21.15.89:0/767253935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a4101e60 0x7f43a4102230 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4394009ce0 tx=0x7f4394004d10 comp rx=0 tx=0).stop 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.052+0000 7f43a9b51700 1 -- 172.21.15.89:0/767253935 shutdown_connections 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.052+0000 7f43a9b51700 1 --2- 172.21.15.89:0/767253935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43a4102800 0x7f43a40fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.052+0000 7f43a9b51700 1 --2- 172.21.15.89:0/767253935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a4101e60 0x7f43a4102230 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.052+0000 7f43a9b51700 1 -- 172.21.15.89:0/767253935 >> 172.21.15.89:0/767253935 conn(0x7f43a40f7a10 msgr2=0x7f43a40f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.052+0000 7f43a9b51700 1 -- 172.21.15.89:0/767253935 shutdown_connections 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.052+0000 7f43a9b51700 1 -- 172.21.15.89:0/767253935 wait complete. 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.052+0000 7f43a9b51700 1 Processor -- start 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.052+0000 7f43a9b51700 1 -- start start 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a9b51700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a4101e60 0x7f43a410c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a9b51700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43a4102800 0x7f43a410ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a9b51700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a410d3d0 con 0x7f43a4101e60 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a9b51700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a410d540 con 0x7f43a4102800 2024-02-10T06:13:42.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a4101e60 0x7f43a410c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:42.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a37fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a4101e60 0x7f43a410c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41264/0 (socket says 172.21.15.89:41264) 2024-02-10T06:13:42.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a37fe700 1 -- 172.21.15.89:0/2744551927 learned_addr learned my addr 172.21.15.89:0/2744551927 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:42.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a37fe700 1 -- 172.21.15.89:0/2744551927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43a4102800 msgr2=0x7f43a410ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:42.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a37fe700 1 --2- 172.21.15.89:0/2744551927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43a4102800 0x7f43a410ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:42.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a37fe700 1 -- 172.21.15.89:0/2744551927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4394009940 con 0x7f43a4101e60 2024-02-10T06:13:42.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a37fe700 1 --2- 172.21.15.89:0/2744551927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a4101e60 0x7f43a410c8c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f439400bf60 tx=0x7f4394004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:42.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f438bfff700 1 -- 172.21.15.89:0/2744551927 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f439400e040 con 0x7f43a4101e60 2024-02-10T06:13:42.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f438bfff700 1 -- 172.21.15.89:0/2744551927 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43940128e0 con 0x7f43a4101e60 2024-02-10T06:13:42.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f43a9b51700 1 -- 172.21.15.89:0/2744551927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a4119f30 con 0x7f43a4101e60 2024-02-10T06:13:42.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.053+0000 7f438bfff700 1 -- 172.21.15.89:0/2744551927 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f439401c870 con 0x7f43a4101e60 2024-02-10T06:13:42.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.054+0000 7f43a9b51700 1 -- 172.21.15.89:0/2744551927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a411a390 con 0x7f43a4101e60 2024-02-10T06:13:42.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.054+0000 7f43a9b51700 1 -- 172.21.15.89:0/2744551927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a40fb610 con 0x7f43a4101e60 2024-02-10T06:13:42.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.055+0000 7f438bfff700 1 -- 172.21.15.89:0/2744551927 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4394012410 con 0x7f43a4101e60 2024-02-10T06:13:42.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.055+0000 7f438bfff700 1 --2- 172.21.15.89:0/2744551927 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f438c0692a0 0x7f438c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:42.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.055+0000 7f438bfff700 1 -- 172.21.15.89:0/2744551927 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4394088b50 con 0x7f43a4101e60 2024-02-10T06:13:42.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.055+0000 7f43a2ffd700 1 --2- 172.21.15.89:0/2744551927 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f438c0692a0 0x7f438c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:42.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.056+0000 7f43a2ffd700 1 --2- 172.21.15.89:0/2744551927 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f438c0692a0 0x7f438c06b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4398006fd0 tx=0x7f4398006db0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:42.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.060+0000 7f438bfff700 1 -- 172.21.15.89:0/2744551927 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4394058bd0 con 0x7f43a4101e60 2024-02-10T06:13:42.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.257+0000 7f43a9b51700 1 -- 172.21.15.89:0/2744551927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f43a411a580 con 0x7f43a4101e60 2024-02-10T06:13:42.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.257+0000 7f438bfff700 1 -- 172.21.15.89:0/2744551927 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4394058760 con 0x7f43a4101e60 2024-02-10T06:13:42.258 INFO:teuthology.orchestra.run.smithi089.stdout:115964117012 2024-02-10T06:13:42.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.259+0000 7f43a9b51700 1 -- 172.21.15.89:0/2744551927 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f438c0692a0 msgr2=0x7f438c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:42.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.259+0000 7f43a9b51700 1 --2- 172.21.15.89:0/2744551927 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f438c0692a0 0x7f438c06b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4398006fd0 tx=0x7f4398006db0 comp rx=0 tx=0).stop 2024-02-10T06:13:42.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.259+0000 7f43a9b51700 1 -- 172.21.15.89:0/2744551927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a4101e60 msgr2=0x7f43a410c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:42.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.259+0000 7f43a9b51700 1 --2- 172.21.15.89:0/2744551927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a4101e60 0x7f43a410c8c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f439400bf60 tx=0x7f4394004d10 comp rx=0 tx=0).stop 2024-02-10T06:13:42.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.259+0000 7f43a9b51700 1 -- 172.21.15.89:0/2744551927 shutdown_connections 2024-02-10T06:13:42.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.259+0000 7f43a9b51700 1 --2- 172.21.15.89:0/2744551927 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f438c0692a0 0x7f438c06b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:42.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.259+0000 7f43a9b51700 1 --2- 172.21.15.89:0/2744551927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43a4101e60 0x7f43a410c8c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:42.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.259+0000 7f43a9b51700 1 --2- 172.21.15.89:0/2744551927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43a4102800 0x7f43a410ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:42.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.259+0000 7f43a9b51700 1 -- 172.21.15.89:0/2744551927 >> 172.21.15.89:0/2744551927 conn(0x7f43a40f7a10 msgr2=0x7f43a40f93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:42.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.259+0000 7f43a9b51700 1 -- 172.21.15.89:0/2744551927 shutdown_connections 2024-02-10T06:13:42.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:42.259+0000 7f43a9b51700 1 -- 172.21.15.89:0/2744551927 wait complete. 2024-02-10T06:13:42.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:42.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:43.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.034+0000 7f4575c81700 1 -- 172.21.15.89:0/937063424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4570101e50 msgr2=0x7f4570102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:43.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.034+0000 7f4575c81700 1 --2- 172.21.15.89:0/937063424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4570101e50 0x7f4570102220 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4560004940 tx=0x7f4560009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:43.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.034+0000 7f4575c81700 1 -- 172.21.15.89:0/937063424 shutdown_connections 2024-02-10T06:13:43.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.034+0000 7f4575c81700 1 --2- 172.21.15.89:0/937063424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45701027f0 0x7f45700fbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:43.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.034+0000 7f4575c81700 1 --2- 172.21.15.89:0/937063424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4570101e50 0x7f4570102220 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:43.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.034+0000 7f4575c81700 1 -- 172.21.15.89:0/937063424 >> 172.21.15.89:0/937063424 conn(0x7f45700f79e0 msgr2=0x7f45700f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:43.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.034+0000 7f4575c81700 1 -- 172.21.15.89:0/937063424 shutdown_connections 2024-02-10T06:13:43.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.034+0000 7f4575c81700 1 -- 172.21.15.89:0/937063424 wait complete. 2024-02-10T06:13:43.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.035+0000 7f4575c81700 1 Processor -- start 2024-02-10T06:13:43.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.035+0000 7f4575c81700 1 -- start start 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.035+0000 7f4575c81700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4570101e50 0x7f4570113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.035+0000 7f4575c81700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45701027f0 0x7f4570113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.035+0000 7f4575c81700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4570114630 con 0x7f45701027f0 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.035+0000 7f4575c81700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457010d960 con 0x7f4570101e50 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.035+0000 7f456effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45701027f0 0x7f4570113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.035+0000 7f456effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45701027f0 0x7f4570113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41286/0 (socket says 172.21.15.89:41286) 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.035+0000 7f456effd700 1 -- 172.21.15.89:0/1488753357 learned_addr learned my addr 172.21.15.89:0/1488753357 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.035+0000 7f456f7fe700 1 --2- 172.21.15.89:0/1488753357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4570101e50 0x7f4570113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.036+0000 7f456effd700 1 -- 172.21.15.89:0/1488753357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4570101e50 msgr2=0x7f4570113aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.036+0000 7f456effd700 1 --2- 172.21.15.89:0/1488753357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4570101e50 0x7f4570113aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.036+0000 7f456effd700 1 -- 172.21.15.89:0/1488753357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4560005040 con 0x7f45701027f0 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.036+0000 7f456effd700 1 --2- 172.21.15.89:0/1488753357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45701027f0 0x7f4570113fe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f456400c800 tx=0x7f4564004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:43.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.036+0000 7f4557fff700 1 -- 172.21.15.89:0/1488753357 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45640106f0 con 0x7f45701027f0 2024-02-10T06:13:43.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.036+0000 7f4575c81700 1 -- 172.21.15.89:0/1488753357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457010dc40 con 0x7f45701027f0 2024-02-10T06:13:43.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.036+0000 7f4557fff700 1 -- 172.21.15.89:0/1488753357 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4564010d20 con 0x7f45701027f0 2024-02-10T06:13:43.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.036+0000 7f4557fff700 1 -- 172.21.15.89:0/1488753357 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45640113a0 con 0x7f45701027f0 2024-02-10T06:13:43.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.036+0000 7f4575c81700 1 -- 172.21.15.89:0/1488753357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457010e180 con 0x7f45701027f0 2024-02-10T06:13:43.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.037+0000 7f4575c81700 1 -- 172.21.15.89:0/1488753357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4570064b00 con 0x7f45701027f0 2024-02-10T06:13:43.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.037+0000 7f4557fff700 1 -- 172.21.15.89:0/1488753357 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4564010850 con 0x7f45701027f0 2024-02-10T06:13:43.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.038+0000 7f4557fff700 1 --2- 172.21.15.89:0/1488753357 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f45580692b0 0x7f455806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:43.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.038+0000 7f4557fff700 1 -- 172.21.15.89:0/1488753357 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4564086e10 con 0x7f45701027f0 2024-02-10T06:13:43.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.038+0000 7f456f7fe700 1 --2- 172.21.15.89:0/1488753357 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f45580692b0 0x7f455806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:43.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.038+0000 7f456f7fe700 1 --2- 172.21.15.89:0/1488753357 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f45580692b0 0x7f455806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f456000b3b0 tx=0x7f4560009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:43.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.042+0000 7f4557fff700 1 -- 172.21.15.89:0/1488753357 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4564056f10 con 0x7f45701027f0 2024-02-10T06:13:43.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.229+0000 7f4575c81700 1 -- 172.21.15.89:0/1488753357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f45700fb5e0 con 0x7f45701027f0 2024-02-10T06:13:43.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.230+0000 7f4557fff700 1 -- 172.21.15.89:0/1488753357 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4564056aa0 con 0x7f45701027f0 2024-02-10T06:13:43.231 INFO:teuthology.orchestra.run.smithi089.stdout:150323855376 2024-02-10T06:13:43.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.232+0000 7f4575c81700 1 -- 172.21.15.89:0/1488753357 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f45580692b0 msgr2=0x7f455806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:43.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.232+0000 7f4575c81700 1 --2- 172.21.15.89:0/1488753357 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f45580692b0 0x7f455806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f456000b3b0 tx=0x7f4560009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:43.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.232+0000 7f4575c81700 1 -- 172.21.15.89:0/1488753357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45701027f0 msgr2=0x7f4570113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:43.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.232+0000 7f4575c81700 1 --2- 172.21.15.89:0/1488753357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45701027f0 0x7f4570113fe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f456400c800 tx=0x7f4564004470 comp rx=0 tx=0).stop 2024-02-10T06:13:43.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.232+0000 7f4575c81700 1 -- 172.21.15.89:0/1488753357 shutdown_connections 2024-02-10T06:13:43.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.232+0000 7f4575c81700 1 --2- 172.21.15.89:0/1488753357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4570101e50 0x7f4570113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:43.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.232+0000 7f4575c81700 1 --2- 172.21.15.89:0/1488753357 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f45580692b0 0x7f455806b760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:43.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.232+0000 7f4575c81700 1 --2- 172.21.15.89:0/1488753357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45701027f0 0x7f4570113fe0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:43.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.232+0000 7f4575c81700 1 -- 172.21.15.89:0/1488753357 >> 172.21.15.89:0/1488753357 conn(0x7f45700f79e0 msgr2=0x7f4570065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:43.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.232+0000 7f4575c81700 1 -- 172.21.15.89:0/1488753357 shutdown_connections 2024-02-10T06:13:43.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:43.232+0000 7f4575c81700 1 -- 172.21.15.89:0/1488753357 wait complete. 2024-02-10T06:13:43.422 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:43 smithi089 ceph-mon[29217]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:43.422 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:43 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2744551927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T06:13:43.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:43 smithi173 ceph-mon[36251]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:43.690 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:43 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2744551927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T06:13:43.739 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117009 got 115964117012 for osd.3 2024-02-10T06:13:43.740 DEBUG:teuthology.parallel:result is None 2024-02-10T06:13:44.107 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:44.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:44 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1488753357' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T06:13:44.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:44 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/1488753357' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T06:13:45.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.335+0000 7f34835f1700 1 -- 172.21.15.89:0/3864300283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f347c102f50 msgr2=0x7f347c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.335+0000 7f34835f1700 1 --2- 172.21.15.89:0/3864300283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f347c102f50 0x7f347c103320 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3478004940 tx=0x7f3478009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.335+0000 7f34835f1700 1 -- 172.21.15.89:0/3864300283 shutdown_connections 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.335+0000 7f34835f1700 1 --2- 172.21.15.89:0/3864300283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f347c1038f0 0x7f347c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.335+0000 7f34835f1700 1 --2- 172.21.15.89:0/3864300283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f347c102f50 0x7f347c103320 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.335+0000 7f34835f1700 1 -- 172.21.15.89:0/3864300283 >> 172.21.15.89:0/3864300283 conn(0x7f347c0f8ae0 msgr2=0x7f347c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.335+0000 7f34835f1700 1 -- 172.21.15.89:0/3864300283 shutdown_connections 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.336+0000 7f34835f1700 1 -- 172.21.15.89:0/3864300283 wait complete. 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.336+0000 7f34835f1700 1 Processor -- start 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.336+0000 7f34835f1700 1 -- start start 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.336+0000 7f34835f1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f347c102f50 0x7f347c113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.336+0000 7f34835f1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f347c1038f0 0x7f347c113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:45.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.336+0000 7f34835f1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c1146c0 con 0x7f347c102f50 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.336+0000 7f34835f1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c10d960 con 0x7f347c1038f0 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.336+0000 7f348138d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f347c102f50 0x7f347c113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.336+0000 7f348138d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f347c102f50 0x7f347c113aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41304/0 (socket says 172.21.15.89:41304) 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.336+0000 7f348138d700 1 -- 172.21.15.89:0/226206977 learned_addr learned my addr 172.21.15.89:0/226206977 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.336+0000 7f3480b8c700 1 --2- 172.21.15.89:0/226206977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f347c1038f0 0x7f347c113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.337+0000 7f348138d700 1 -- 172.21.15.89:0/226206977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f347c1038f0 msgr2=0x7f347c113fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.337+0000 7f348138d700 1 --2- 172.21.15.89:0/226206977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f347c1038f0 0x7f347c113fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.337+0000 7f348138d700 1 -- 172.21.15.89:0/226206977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3478005040 con 0x7f347c102f50 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.337+0000 7f348138d700 1 --2- 172.21.15.89:0/226206977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f347c102f50 0x7f347c113aa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f347800b3b0 tx=0x7f3478009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.337+0000 7f3471ffb700 1 -- 172.21.15.89:0/226206977 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f347800e040 con 0x7f347c102f50 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.337+0000 7f3471ffb700 1 -- 172.21.15.89:0/226206977 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f347800bbf0 con 0x7f347c102f50 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.337+0000 7f34835f1700 1 -- 172.21.15.89:0/226206977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c10dbe0 con 0x7f347c102f50 2024-02-10T06:13:45.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.337+0000 7f3471ffb700 1 -- 172.21.15.89:0/226206977 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f347801c3f0 con 0x7f347c102f50 2024-02-10T06:13:45.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.337+0000 7f34835f1700 1 -- 172.21.15.89:0/226206977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c10e120 con 0x7f347c102f50 2024-02-10T06:13:45.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.338+0000 7f34835f1700 1 -- 172.21.15.89:0/226206977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347c0fc6e0 con 0x7f347c102f50 2024-02-10T06:13:45.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.338+0000 7f3471ffb700 1 -- 172.21.15.89:0/226206977 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3478013420 con 0x7f347c102f50 2024-02-10T06:13:45.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.339+0000 7f3471ffb700 1 --2- 172.21.15.89:0/226206977 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f34680692a0 0x7f346806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:45.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.339+0000 7f3471ffb700 1 -- 172.21.15.89:0/226206977 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3478088190 con 0x7f347c102f50 2024-02-10T06:13:45.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.339+0000 7f3480b8c700 1 --2- 172.21.15.89:0/226206977 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f34680692a0 0x7f346806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:45.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.339+0000 7f3480b8c700 1 --2- 172.21.15.89:0/226206977 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f34680692a0 0x7f346806b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f347c10f2d0 tx=0x7f346c0091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:45.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.343+0000 7f3471ffb700 1 -- 172.21.15.89:0/226206977 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3478058290 con 0x7f347c102f50 2024-02-10T06:13:45.429 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:45 smithi089 ceph-mon[29217]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:45.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.536+0000 7f34835f1700 1 -- 172.21.15.89:0/226206977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f347c10f080 con 0x7f347c102f50 2024-02-10T06:13:45.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.536+0000 7f3471ffb700 1 -- 172.21.15.89:0/226206977 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3478057e20 con 0x7f347c102f50 2024-02-10T06:13:45.537 INFO:teuthology.orchestra.run.smithi089.stdout:38654705700 2024-02-10T06:13:45.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.538+0000 7f34835f1700 1 -- 172.21.15.89:0/226206977 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f34680692a0 msgr2=0x7f346806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.538+0000 7f34835f1700 1 --2- 172.21.15.89:0/226206977 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f34680692a0 0x7f346806b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f347c10f2d0 tx=0x7f346c0091a0 comp rx=0 tx=0).stop 2024-02-10T06:13:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.538+0000 7f34835f1700 1 -- 172.21.15.89:0/226206977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f347c102f50 msgr2=0x7f347c113aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.538+0000 7f34835f1700 1 --2- 172.21.15.89:0/226206977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f347c102f50 0x7f347c113aa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f347800b3b0 tx=0x7f3478009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.538+0000 7f34835f1700 1 -- 172.21.15.89:0/226206977 shutdown_connections 2024-02-10T06:13:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.538+0000 7f34835f1700 1 --2- 172.21.15.89:0/226206977 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f34680692a0 0x7f346806b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.538+0000 7f34835f1700 1 --2- 172.21.15.89:0/226206977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f347c102f50 0x7f347c113aa0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.538+0000 7f34835f1700 1 --2- 172.21.15.89:0/226206977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f347c1038f0 0x7f347c113fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.538+0000 7f34835f1700 1 -- 172.21.15.89:0/226206977 >> 172.21.15.89:0/226206977 conn(0x7f347c0f8ae0 msgr2=0x7f347c0fa460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.538+0000 7f34835f1700 1 -- 172.21.15.89:0/226206977 shutdown_connections 2024-02-10T06:13:45.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:45.538+0000 7f34835f1700 1 -- 172.21.15.89:0/226206977 wait complete. 2024-02-10T06:13:45.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:45 smithi173 ceph-mon[36251]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:46.151 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855374 got 150323855376 for osd.4 2024-02-10T06:13:46.151 DEBUG:teuthology.parallel:result is None 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.360+0000 7fabfd11c700 1 -- 172.21.15.89:0/1422650277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf81038f0 msgr2=0x7fabf80fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.360+0000 7fabfd11c700 1 --2- 172.21.15.89:0/1422650277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf81038f0 0x7fabf80fcf60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fabe0004940 tx=0x7fabe0009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.360+0000 7fabfd11c700 1 -- 172.21.15.89:0/1422650277 shutdown_connections 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.360+0000 7fabfd11c700 1 --2- 172.21.15.89:0/1422650277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf81038f0 0x7fabf80fcf60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.360+0000 7fabfd11c700 1 --2- 172.21.15.89:0/1422650277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabf8102f50 0x7fabf8103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.360+0000 7fabfd11c700 1 -- 172.21.15.89:0/1422650277 >> 172.21.15.89:0/1422650277 conn(0x7fabf80f8ae0 msgr2=0x7fabf80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.360+0000 7fabfd11c700 1 -- 172.21.15.89:0/1422650277 shutdown_connections 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.360+0000 7fabfd11c700 1 -- 172.21.15.89:0/1422650277 wait complete. 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.360+0000 7fabfd11c700 1 Processor -- start 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabfd11c700 1 -- start start 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabfd11c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabf8102f50 0x7fabf8070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabfd11c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf81038f0 0x7fabf80707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:46.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabfd11c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8070dd0 con 0x7fabf81038f0 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabfd11c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8070f40 con 0x7fabf8102f50 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabee59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf81038f0 0x7fabf80707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabee59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf81038f0 0x7fabf80707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41324/0 (socket says 172.21.15.89:41324) 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabee59c700 1 -- 172.21.15.89:0/2363454112 learned_addr learned my addr 172.21.15.89:0/2363454112 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabee59c700 1 -- 172.21.15.89:0/2363454112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabf8102f50 msgr2=0x7fabf8070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabf6d9d700 1 --2- 172.21.15.89:0/2363454112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabf8102f50 0x7fabf8070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabee59c700 1 --2- 172.21.15.89:0/2363454112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabf8102f50 0x7fabf8070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabee59c700 1 -- 172.21.15.89:0/2363454112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe0005040 con 0x7fabf81038f0 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.361+0000 7fabee59c700 1 --2- 172.21.15.89:0/2363454112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf81038f0 0x7fabf80707d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009150 tx=0x7fabe0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.362+0000 7fabeffff700 1 -- 172.21.15.89:0/2363454112 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabe000e040 con 0x7fabf81038f0 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.362+0000 7fabeffff700 1 -- 172.21.15.89:0/2363454112 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabe000bbf0 con 0x7fabf81038f0 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.362+0000 7fabeffff700 1 -- 172.21.15.89:0/2363454112 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabe001c630 con 0x7fabf81038f0 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.362+0000 7fabfd11c700 1 -- 172.21.15.89:0/2363454112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf8119cb0 con 0x7fabf81038f0 2024-02-10T06:13:46.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.362+0000 7fabfd11c700 1 -- 172.21.15.89:0/2363454112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf811a110 con 0x7fabf81038f0 2024-02-10T06:13:46.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.362+0000 7fabfd11c700 1 -- 172.21.15.89:0/2363454112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf8105420 con 0x7fabf81038f0 2024-02-10T06:13:46.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.363+0000 7fabeffff700 1 -- 172.21.15.89:0/2363454112 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fabe000bd60 con 0x7fabf81038f0 2024-02-10T06:13:46.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.363+0000 7fabeffff700 1 --2- 172.21.15.89:0/2363454112 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fabe40692a0 0x7fabe406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:46.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.363+0000 7fabeffff700 1 -- 172.21.15.89:0/2363454112 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fabe0088110 con 0x7fabf81038f0 2024-02-10T06:13:46.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.363+0000 7fabf6d9d700 1 --2- 172.21.15.89:0/2363454112 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fabe40692a0 0x7fabe406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:46.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.364+0000 7fabf6d9d700 1 --2- 172.21.15.89:0/2363454112 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fabe40692a0 0x7fabe406b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fabe8004c40 tx=0x7fabe8009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:46.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.368+0000 7fabeffff700 1 -- 172.21.15.89:0/2363454112 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fabe0054820 con 0x7fabf81038f0 2024-02-10T06:13:46.380 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:46 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/226206977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T06:13:46.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.554+0000 7fabfd11c700 1 -- 172.21.15.89:0/2363454112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fabf80fc6e0 con 0x7fabf81038f0 2024-02-10T06:13:46.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.555+0000 7fabeffff700 1 -- 172.21.15.89:0/2363454112 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fabe0057e40 con 0x7fabf81038f0 2024-02-10T06:13:46.557 INFO:teuthology.orchestra.run.smithi089.stdout:60129542176 2024-02-10T06:13:46.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.557+0000 7fabfd11c700 1 -- 172.21.15.89:0/2363454112 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fabe40692a0 msgr2=0x7fabe406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:46.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.557+0000 7fabfd11c700 1 --2- 172.21.15.89:0/2363454112 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fabe40692a0 0x7fabe406b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fabe8004c40 tx=0x7fabe8009040 comp rx=0 tx=0).stop 2024-02-10T06:13:46.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.557+0000 7fabfd11c700 1 -- 172.21.15.89:0/2363454112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf81038f0 msgr2=0x7fabf80707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:46.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.557+0000 7fabfd11c700 1 --2- 172.21.15.89:0/2363454112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf81038f0 0x7fabf80707d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009150 tx=0x7fabe0009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.557+0000 7fabfd11c700 1 -- 172.21.15.89:0/2363454112 shutdown_connections 2024-02-10T06:13:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.557+0000 7fabfd11c700 1 --2- 172.21.15.89:0/2363454112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabf8102f50 0x7fabf8070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.557+0000 7fabfd11c700 1 --2- 172.21.15.89:0/2363454112 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fabe40692a0 0x7fabe406b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.557+0000 7fabfd11c700 1 --2- 172.21.15.89:0/2363454112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf81038f0 0x7fabf80707d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.557+0000 7fabfd11c700 1 -- 172.21.15.89:0/2363454112 >> 172.21.15.89:0/2363454112 conn(0x7fabf80f8ae0 msgr2=0x7fabf80fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:46.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.557+0000 7fabfd11c700 1 -- 172.21.15.89:0/2363454112 shutdown_connections 2024-02-10T06:13:46.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:46.557+0000 7fabfd11c700 1 -- 172.21.15.89:0/2363454112 wait complete. 2024-02-10T06:13:46.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:46 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/226206977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T06:13:47.243 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705698 got 38654705700 for osd.0 2024-02-10T06:13:47.243 DEBUG:teuthology.parallel:result is None 2024-02-10T06:13:47.603 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:47 smithi089 ceph-mon[29217]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:47.603 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:47 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2363454112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T06:13:47.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:47 smithi173 ceph-mon[36251]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:47.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:47 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2363454112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T06:13:47.974 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542176 for osd.1 2024-02-10T06:13:47.974 DEBUG:teuthology.parallel:result is None 2024-02-10T06:13:48.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.427+0000 7f0e1c781700 1 -- 172.21.15.89:0/1497404308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e14101e40 msgr2=0x7f0e14102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:48.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.427+0000 7f0e1c781700 1 --2- 172.21.15.89:0/1497404308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e14101e40 0x7f0e14102210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0e10004940 tx=0x7f0e10009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:48.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.427+0000 7f0e1c781700 1 -- 172.21.15.89:0/1497404308 shutdown_connections 2024-02-10T06:13:48.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.427+0000 7f0e1c781700 1 --2- 172.21.15.89:0/1497404308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e141027e0 0x7f0e140fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:48.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.427+0000 7f0e1c781700 1 --2- 172.21.15.89:0/1497404308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e14101e40 0x7f0e14102210 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:48.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.427+0000 7f0e1c781700 1 -- 172.21.15.89:0/1497404308 >> 172.21.15.89:0/1497404308 conn(0x7f0e14071d90 msgr2=0x7f0e14072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:48.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.427+0000 7f0e1c781700 1 -- 172.21.15.89:0/1497404308 shutdown_connections 2024-02-10T06:13:48.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.427+0000 7f0e1c781700 1 -- 172.21.15.89:0/1497404308 wait complete. 2024-02-10T06:13:48.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.427+0000 7f0e1c781700 1 Processor -- start 2024-02-10T06:13:48.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.427+0000 7f0e1c781700 1 -- start start 2024-02-10T06:13:48.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1c781700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e14101e40 0x7f0e14112970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:48.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1c781700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e141027e0 0x7f0e14112eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:48.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1c781700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e14113590 con 0x7f0e14101e40 2024-02-10T06:13:48.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1c781700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1410c830 con 0x7f0e141027e0 2024-02-10T06:13:48.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1a51d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e14101e40 0x7f0e14112970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:48.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1a51d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e14101e40 0x7f0e14112970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41344/0 (socket says 172.21.15.89:41344) 2024-02-10T06:13:48.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1a51d700 1 -- 172.21.15.89:0/51746326 learned_addr learned my addr 172.21.15.89:0/51746326 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:48.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1a51d700 1 -- 172.21.15.89:0/51746326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e141027e0 msgr2=0x7f0e14112eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:48.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1a51d700 1 --2- 172.21.15.89:0/51746326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e141027e0 0x7f0e14112eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:48.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1a51d700 1 -- 172.21.15.89:0/51746326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e10005040 con 0x7f0e14101e40 2024-02-10T06:13:48.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1a51d700 1 --2- 172.21.15.89:0/51746326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e14101e40 0x7f0e14112970 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000b3b0 tx=0x7f0e10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:48.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e0affd700 1 -- 172.21.15.89:0/51746326 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e1000e040 con 0x7f0e14101e40 2024-02-10T06:13:48.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1c781700 1 -- 172.21.15.89:0/51746326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1410cab0 con 0x7f0e14101e40 2024-02-10T06:13:48.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e1c781700 1 -- 172.21.15.89:0/51746326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1410cff0 con 0x7f0e14101e40 2024-02-10T06:13:48.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e0affd700 1 -- 172.21.15.89:0/51746326 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e1000bbf0 con 0x7f0e14101e40 2024-02-10T06:13:48.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.428+0000 7f0e0affd700 1 -- 172.21.15.89:0/51746326 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e1001c630 con 0x7f0e14101e40 2024-02-10T06:13:48.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.429+0000 7f0e1c781700 1 -- 172.21.15.89:0/51746326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e140738b0 con 0x7f0e14101e40 2024-02-10T06:13:48.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.430+0000 7f0e0affd700 1 -- 172.21.15.89:0/51746326 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0e1000bd60 con 0x7f0e14101e40 2024-02-10T06:13:48.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.430+0000 7f0e0affd700 1 --2- 172.21.15.89:0/51746326 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e000692a0 0x7f0e0006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:48.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.430+0000 7f0e0affd700 1 -- 172.21.15.89:0/51746326 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0e10088170 con 0x7f0e14101e40 2024-02-10T06:13:48.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.430+0000 7f0e19d1c700 1 --2- 172.21.15.89:0/51746326 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e000692a0 0x7f0e0006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:48.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.431+0000 7f0e19d1c700 1 --2- 172.21.15.89:0/51746326 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e000692a0 0x7f0e0006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0e04006fd0 tx=0x7f0e04006db0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:48.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.435+0000 7f0e0affd700 1 -- 172.21.15.89:0/51746326 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0e10058310 con 0x7f0e14101e40 2024-02-10T06:13:48.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.622+0000 7f0e1c781700 1 -- 172.21.15.89:0/51746326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0e1404bf00 con 0x7f0e14101e40 2024-02-10T06:13:48.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.622+0000 7f0e0affd700 1 -- 172.21.15.89:0/51746326 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0e10057ea0 con 0x7f0e14101e40 2024-02-10T06:13:48.623 INFO:teuthology.orchestra.run.smithi089.stdout:171798691852 2024-02-10T06:13:48.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.624+0000 7f0e1c781700 1 -- 172.21.15.89:0/51746326 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e000692a0 msgr2=0x7f0e0006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:48.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.624+0000 7f0e1c781700 1 --2- 172.21.15.89:0/51746326 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e000692a0 0x7f0e0006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0e04006fd0 tx=0x7f0e04006db0 comp rx=0 tx=0).stop 2024-02-10T06:13:48.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.624+0000 7f0e1c781700 1 -- 172.21.15.89:0/51746326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e14101e40 msgr2=0x7f0e14112970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:48.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.624+0000 7f0e1c781700 1 --2- 172.21.15.89:0/51746326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e14101e40 0x7f0e14112970 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000b3b0 tx=0x7f0e10009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:48.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.624+0000 7f0e1c781700 1 -- 172.21.15.89:0/51746326 shutdown_connections 2024-02-10T06:13:48.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.624+0000 7f0e1c781700 1 --2- 172.21.15.89:0/51746326 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e000692a0 0x7f0e0006b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:48.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.624+0000 7f0e1c781700 1 --2- 172.21.15.89:0/51746326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e14101e40 0x7f0e14112970 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:48.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.624+0000 7f0e1c781700 1 --2- 172.21.15.89:0/51746326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e141027e0 0x7f0e14112eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:48.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.624+0000 7f0e1c781700 1 -- 172.21.15.89:0/51746326 >> 172.21.15.89:0/51746326 conn(0x7f0e14071d90 msgr2=0x7f0e140fad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:48.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.624+0000 7f0e1c781700 1 -- 172.21.15.89:0/51746326 shutdown_connections 2024-02-10T06:13:48.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:48.624+0000 7f0e1c781700 1 -- 172.21.15.89:0/51746326 wait complete. 2024-02-10T06:13:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.188+0000 7fa0e55e1700 1 -- 172.21.15.89:0/1288863869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e01030c0 msgr2=0x7fa0e0103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.188+0000 7fa0e55e1700 1 --2- 172.21.15.89:0/1288863869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e01030c0 0x7fa0e0103490 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0004940 tx=0x7fa0d0009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.188+0000 7fa0e55e1700 1 -- 172.21.15.89:0/1288863869 shutdown_connections 2024-02-10T06:13:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.188+0000 7fa0e55e1700 1 --2- 172.21.15.89:0/1288863869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0e00fb560 0x7fa0e00fb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:49.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.188+0000 7fa0e55e1700 1 --2- 172.21.15.89:0/1288863869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e01030c0 0x7fa0e0103490 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:49.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.188+0000 7fa0e55e1700 1 -- 172.21.15.89:0/1288863869 >> 172.21.15.89:0/1288863869 conn(0x7fa0e0071500 msgr2=0x7fa0e0071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:49.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.188+0000 7fa0e55e1700 1 -- 172.21.15.89:0/1288863869 shutdown_connections 2024-02-10T06:13:49.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.188+0000 7fa0e55e1700 1 -- 172.21.15.89:0/1288863869 wait complete. 2024-02-10T06:13:49.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0e55e1700 1 Processor -- start 2024-02-10T06:13:49.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0e55e1700 1 -- start start 2024-02-10T06:13:49.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0e55e1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0e00fb560 0x7fa0e0113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:49.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0e55e1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e01030c0 0x7fa0e0113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:49.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0e55e1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e0114630 con 0x7fa0e01030c0 2024-02-10T06:13:49.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0e55e1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e010d960 con 0x7fa0e00fb560 2024-02-10T06:13:49.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0de7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e01030c0 0x7fa0e0113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:49.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0de7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e01030c0 0x7fa0e0113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41590/0 (socket says 172.21.15.89:41590) 2024-02-10T06:13:49.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0de7fc700 1 -- 172.21.15.89:0/3628438954 learned_addr learned my addr 172.21.15.89:0/3628438954 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:49.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0deffd700 1 --2- 172.21.15.89:0/3628438954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0e00fb560 0x7fa0e0113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:49.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0de7fc700 1 -- 172.21.15.89:0/3628438954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0e00fb560 msgr2=0x7fa0e0113aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:49.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0de7fc700 1 --2- 172.21.15.89:0/3628438954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0e00fb560 0x7fa0e0113aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:49.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.189+0000 7fa0de7fc700 1 -- 172.21.15.89:0/3628438954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d0005040 con 0x7fa0e01030c0 2024-02-10T06:13:49.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.190+0000 7fa0de7fc700 1 --2- 172.21.15.89:0/3628438954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e01030c0 0x7fa0e0113fe0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4001d80 tx=0x7fa0d4009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:49.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.190+0000 7fa0c77fe700 1 -- 172.21.15.89:0/3628438954 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d4007d10 con 0x7fa0e01030c0 2024-02-10T06:13:49.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.190+0000 7fa0c77fe700 1 -- 172.21.15.89:0/3628438954 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0d4001e80 con 0x7fa0e01030c0 2024-02-10T06:13:49.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.190+0000 7fa0e55e1700 1 -- 172.21.15.89:0/3628438954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e010dc40 con 0x7fa0e01030c0 2024-02-10T06:13:49.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.190+0000 7fa0c77fe700 1 -- 172.21.15.89:0/3628438954 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d4014640 con 0x7fa0e01030c0 2024-02-10T06:13:49.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.190+0000 7fa0e55e1700 1 -- 172.21.15.89:0/3628438954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e010e180 con 0x7fa0e01030c0 2024-02-10T06:13:49.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.191+0000 7fa0e55e1700 1 -- 172.21.15.89:0/3628438954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e0100290 con 0x7fa0e01030c0 2024-02-10T06:13:49.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.191+0000 7fa0c77fe700 1 -- 172.21.15.89:0/3628438954 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa0d400e960 con 0x7fa0e01030c0 2024-02-10T06:13:49.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.191+0000 7fa0c77fe700 1 --2- 172.21.15.89:0/3628438954 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fa0c80692a0 0x7fa0c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:49.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.192+0000 7fa0c77fe700 1 -- 172.21.15.89:0/3628438954 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa0d4086bc0 con 0x7fa0e01030c0 2024-02-10T06:13:49.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.192+0000 7fa0deffd700 1 --2- 172.21.15.89:0/3628438954 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fa0c80692a0 0x7fa0c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:49.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.192+0000 7fa0deffd700 1 --2- 172.21.15.89:0/3628438954 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fa0c80692a0 0x7fa0c806b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0004f90 tx=0x7fa0d0009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:49.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.196+0000 7fa0c77fe700 1 -- 172.21.15.89:0/3628438954 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa0d404dbd0 con 0x7fa0e01030c0 2024-02-10T06:13:49.309 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:48 smithi089 ceph-mon[29217]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:49.309 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:48 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/51746326' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T06:13:49.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.382+0000 7fa0e55e1700 1 -- 172.21.15.89:0/3628438954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa0e010a130 con 0x7fa0e01030c0 2024-02-10T06:13:49.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.382+0000 7fa0c77fe700 1 -- 172.21.15.89:0/3628438954 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa0d4056cf0 con 0x7fa0e01030c0 2024-02-10T06:13:49.383 INFO:teuthology.orchestra.run.smithi089.stdout:81604378651 2024-02-10T06:13:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.384+0000 7fa0e55e1700 1 -- 172.21.15.89:0/3628438954 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fa0c80692a0 msgr2=0x7fa0c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.384+0000 7fa0e55e1700 1 --2- 172.21.15.89:0/3628438954 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fa0c80692a0 0x7fa0c806b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0004f90 tx=0x7fa0d0009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.384+0000 7fa0e55e1700 1 -- 172.21.15.89:0/3628438954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e01030c0 msgr2=0x7fa0e0113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.384+0000 7fa0e55e1700 1 --2- 172.21.15.89:0/3628438954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e01030c0 0x7fa0e0113fe0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4001d80 tx=0x7fa0d4009ec0 comp rx=0 tx=0).stop 2024-02-10T06:13:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.384+0000 7fa0e55e1700 1 -- 172.21.15.89:0/3628438954 shutdown_connections 2024-02-10T06:13:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.384+0000 7fa0e55e1700 1 --2- 172.21.15.89:0/3628438954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0e00fb560 0x7fa0e0113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.384+0000 7fa0e55e1700 1 --2- 172.21.15.89:0/3628438954 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fa0c80692a0 0x7fa0c806b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.384+0000 7fa0e55e1700 1 --2- 172.21.15.89:0/3628438954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0e01030c0 0x7fa0e0113fe0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:49.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.384+0000 7fa0e55e1700 1 -- 172.21.15.89:0/3628438954 >> 172.21.15.89:0/3628438954 conn(0x7fa0e0071500 msgr2=0x7fa0e00fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.384+0000 7fa0e55e1700 1 -- 172.21.15.89:0/3628438954 shutdown_connections 2024-02-10T06:13:49.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:49.384+0000 7fa0e55e1700 1 -- 172.21.15.89:0/3628438954 wait complete. 2024-02-10T06:13:49.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:48 smithi173 ceph-mon[36251]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:49.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:48 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/51746326' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T06:13:49.717 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691849 got 171798691852 for osd.5 2024-02-10T06:13:49.717 DEBUG:teuthology.parallel:result is None 2024-02-10T06:13:50.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:49 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3628438954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T06:13:50.382 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378651 for osd.2 2024-02-10T06:13:50.382 DEBUG:teuthology.parallel:result is None 2024-02-10T06:13:50.382 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T06:13:50.383 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T06:13:50.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:49 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/3628438954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T06:13:50.639 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:51.345 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:51 smithi089 ceph-mon[29217]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:51.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:51 smithi173 ceph-mon[36251]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.181+0000 7fefad438700 1 -- 172.21.15.89:0/4097794950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefa80fdf50 msgr2=0x7fefa80fe320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.181+0000 7fefad438700 1 --2- 172.21.15.89:0/4097794950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefa80fdf50 0x7fefa80fe320 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fef98004940 tx=0x7fef98009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.181+0000 7fefad438700 1 -- 172.21.15.89:0/4097794950 shutdown_connections 2024-02-10T06:13:52.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.181+0000 7fefad438700 1 --2- 172.21.15.89:0/4097794950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefa80fe8f0 0x7fefa8106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.181+0000 7fefad438700 1 --2- 172.21.15.89:0/4097794950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefa80fdf50 0x7fefa80fe320 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.181+0000 7fefad438700 1 -- 172.21.15.89:0/4097794950 >> 172.21.15.89:0/4097794950 conn(0x7fefa80f79e0 msgr2=0x7fefa80f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.181+0000 7fefad438700 1 -- 172.21.15.89:0/4097794950 shutdown_connections 2024-02-10T06:13:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.181+0000 7fefad438700 1 -- 172.21.15.89:0/4097794950 wait complete. 2024-02-10T06:13:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fefad438700 1 Processor -- start 2024-02-10T06:13:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fefad438700 1 -- start start 2024-02-10T06:13:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fefad438700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefa80fdf50 0x7fefa8115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fefad438700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefa80fe8f0 0x7fefa8116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fefad438700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa8116820 con 0x7fefa80fe8f0 2024-02-10T06:13:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fefad438700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa810fae0 con 0x7fefa80fdf50 2024-02-10T06:13:52.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fef9e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefa80fe8f0 0x7fefa8116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fef9e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefa80fe8f0 0x7fefa8116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41614/0 (socket says 172.21.15.89:41614) 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fef9e7fc700 1 -- 172.21.15.89:0/2565583305 learned_addr learned my addr 172.21.15.89:0/2565583305 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fefa6ffd700 1 --2- 172.21.15.89:0/2565583305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefa80fdf50 0x7fefa8115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fef9e7fc700 1 -- 172.21.15.89:0/2565583305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefa80fdf50 msgr2=0x7fefa8115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fef9e7fc700 1 --2- 172.21.15.89:0/2565583305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefa80fdf50 0x7fefa8115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.182+0000 7fef9e7fc700 1 -- 172.21.15.89:0/2565583305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef98005040 con 0x7fefa80fe8f0 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.183+0000 7fef9e7fc700 1 --2- 172.21.15.89:0/2565583305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefa80fe8f0 0x7fefa8116140 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fef9000b790 tx=0x7fef90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.183+0000 7fef9ffff700 1 -- 172.21.15.89:0/2565583305 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9000f770 con 0x7fefa80fe8f0 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.183+0000 7fef9ffff700 1 -- 172.21.15.89:0/2565583305 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef9000fda0 con 0x7fefa80fe8f0 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.183+0000 7fefad438700 1 -- 172.21.15.89:0/2565583305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa810fdc0 con 0x7fefa80fe8f0 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.183+0000 7fef9ffff700 1 -- 172.21.15.89:0/2565583305 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9000d310 con 0x7fefa80fe8f0 2024-02-10T06:13:52.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.183+0000 7fefad438700 1 -- 172.21.15.89:0/2565583305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa8110300 con 0x7fefa80fe8f0 2024-02-10T06:13:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.184+0000 7fefad438700 1 -- 172.21.15.89:0/2565583305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa8106940 con 0x7fefa80fe8f0 2024-02-10T06:13:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.184+0000 7fef9ffff700 1 -- 172.21.15.89:0/2565583305 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fef90010420 con 0x7fefa80fe8f0 2024-02-10T06:13:52.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.184+0000 7fef9ffff700 1 --2- 172.21.15.89:0/2565583305 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fef940692a0 0x7fef9406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.185+0000 7fef9ffff700 1 -- 172.21.15.89:0/2565583305 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fef90086e70 con 0x7fefa80fe8f0 2024-02-10T06:13:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.185+0000 7fefa6ffd700 1 --2- 172.21.15.89:0/2565583305 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fef940692a0 0x7fef9406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:52.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.185+0000 7fefa6ffd700 1 --2- 172.21.15.89:0/2565583305 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fef940692a0 0x7fef9406b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fef98004f90 tx=0x7fef98009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:52.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.189+0000 7fef9ffff700 1 -- 172.21.15.89:0/2565583305 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fef900534e0 con 0x7fefa80fe8f0 2024-02-10T06:13:52.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.367+0000 7fefad438700 1 -- 172.21.15.89:0/2565583305 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fefa804bed0 con 0x7fef940692a0 2024-02-10T06:13:52.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.368+0000 7fef9ffff700 1 -- 172.21.15.89:0/2565583305 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27643 (secure 0 0 0) 0x7fefa804bed0 con 0x7fef940692a0 2024-02-10T06:13:52.369 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:13:52.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.370+0000 7fefad438700 1 -- 172.21.15.89:0/2565583305 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fef940692a0 msgr2=0x7fef9406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:52.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.370+0000 7fefad438700 1 --2- 172.21.15.89:0/2565583305 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fef940692a0 0x7fef9406b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fef98004f90 tx=0x7fef98009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:52.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.370+0000 7fefad438700 1 -- 172.21.15.89:0/2565583305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefa80fe8f0 msgr2=0x7fefa8116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:52.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.370+0000 7fefad438700 1 --2- 172.21.15.89:0/2565583305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefa80fe8f0 0x7fefa8116140 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fef9000b790 tx=0x7fef90009250 comp rx=0 tx=0).stop 2024-02-10T06:13:52.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.370+0000 7fefad438700 1 -- 172.21.15.89:0/2565583305 shutdown_connections 2024-02-10T06:13:52.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.370+0000 7fefad438700 1 --2- 172.21.15.89:0/2565583305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefa80fdf50 0x7fefa8115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:52.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.370+0000 7fefad438700 1 --2- 172.21.15.89:0/2565583305 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fef940692a0 0x7fef9406b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:52.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.370+0000 7fefad438700 1 --2- 172.21.15.89:0/2565583305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefa80fe8f0 0x7fefa8116140 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:52.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.370+0000 7fefad438700 1 -- 172.21.15.89:0/2565583305 >> 172.21.15.89:0/2565583305 conn(0x7fefa80f79e0 msgr2=0x7fefa8103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:52.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.370+0000 7fefad438700 1 -- 172.21.15.89:0/2565583305 shutdown_connections 2024-02-10T06:13:52.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:52.371+0000 7fefad438700 1 -- 172.21.15.89:0/2565583305 wait complete. 2024-02-10T06:13:52.372 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-02-10T06:13:53.087 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-02-10T06:13:52.136569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002145"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-10T06:13:03.324896+0000","last_change":"2024-02-10T06:12:38.834570+0000","last_active":"2024-02-10T06:13:03.324896+0000","last_peered":"2024-02-10T06:13:03.324896+0000","last_clean":"2024-02-10T06:13:03.324896+0000","last_became_active":"2024-02-10T06:12:38.834320+0000","last_became_peered":"2024-02-10T06:12:38.834320+0000","last_unstale":"2024-02-10T06:13:03.324896+0000","last_undegraded":"2024-02-10T06:13:03.324896+0000","last_fullsized":"2024-02-10T06:13:03.324896+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T06:11:52.993144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T06:11:52.993144+0000","last_clean_scrub_stamp":"2024-02-10T06:11:52.993144+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:16:19.347522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.46500000000000002}]},{"osd":1,"last update":"Sat Feb 10 06:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Feb 10 06:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Feb 10 06:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.40000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":6.5780000000000003,"5min":6.5780000000000003,"15min":6.5780000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.495}]},{"osd":1,"last update":"Sat Feb 10 06:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.6440000000000001,"5min":6.6440000000000001,"15min":6.6440000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.36399999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.617,"5min":6.617,"15min":6.617},"last":0.52000000000000002}]},{"osd":4,"last update":"Sat Feb 10 06:13:40 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.437,"5min":0.437,"15min":0.437},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.45200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.30399999999999999}]},{"osd":1,"last update":"Sat Feb 10 06:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.33400000000000002}]},{"osd":3,"last update":"Sat Feb 10 06:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.63700000000000001}]},{"osd":4,"last update":"Sat Feb 10 06:13:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705701,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 06:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.312,"15min":0.30499999999999999},"min":{"1min":0.22,"5min":0.154,"15min":0.154},"max":{"1min":0.44600000000000001,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.31,"15min":0.30299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.50700000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.434,"5min":0.434,"15min":0.434},"last":0.245},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.26500000000000001}]},{"osd":3,"last update":"Sat Feb 10 06:13:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.59499999999999997}]},{"osd":4,"last update":"Sat Feb 10 06:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:13:27 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.251,"5min":0.218,"15min":0.218},"max":{"1min":0.41999999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.34000000000000002,"15min":0.33400000000000002},"min":{"1min":0.247,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.5,"5min":0.503,"15min":0.503},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:12:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Feb 10 06:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":14.904,"5min":14.904,"15min":14.904},"last":0.497}]},{"osd":4,"last update":"Sat Feb 10 06:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.42899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T06:13:53.088 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T06:13:53.344 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:53.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:53 smithi089 ceph-mon[29217]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:53.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:53 smithi089 ceph-mon[29217]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:13:53.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:53 smithi173 ceph-mon[36251]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:53.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:53 smithi173 ceph-mon[36251]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:13:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.819+0000 7f2f2e55d700 1 -- 172.21.15.89:0/1809108656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f28102800 msgr2=0x7f2f280fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.819+0000 7f2f2e55d700 1 --2- 172.21.15.89:0/1809108656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f28102800 0x7f2f280fbe70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2f18004940 tx=0x7f2f18009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.819+0000 7f2f2e55d700 1 -- 172.21.15.89:0/1809108656 shutdown_connections 2024-02-10T06:13:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.819+0000 7f2f2e55d700 1 --2- 172.21.15.89:0/1809108656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f28102800 0x7f2f280fbe70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.819+0000 7f2f2e55d700 1 --2- 172.21.15.89:0/1809108656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f28101e60 0x7f2f28102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.819+0000 7f2f2e55d700 1 -- 172.21.15.89:0/1809108656 >> 172.21.15.89:0/1809108656 conn(0x7f2f280f7a10 msgr2=0x7f2f280f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.819+0000 7f2f2e55d700 1 -- 172.21.15.89:0/1809108656 shutdown_connections 2024-02-10T06:13:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.819+0000 7f2f2e55d700 1 -- 172.21.15.89:0/1809108656 wait complete. 2024-02-10T06:13:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.820+0000 7f2f2e55d700 1 Processor -- start 2024-02-10T06:13:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.820+0000 7f2f2e55d700 1 -- start start 2024-02-10T06:13:54.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.820+0000 7f2f2e55d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f28101e60 0x7f2f2810eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.820+0000 7f2f2e55d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f28102800 0x7f2f2810f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.820+0000 7f2f2e55d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2810f670 con 0x7f2f28101e60 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.820+0000 7f2f2e55d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2810f7b0 con 0x7f2f28102800 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.820+0000 7f2f27fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f28101e60 0x7f2f2810eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.820+0000 7f2f27fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f28101e60 0x7f2f2810eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41630/0 (socket says 172.21.15.89:41630) 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.820+0000 7f2f27fff700 1 -- 172.21.15.89:0/3238169125 learned_addr learned my addr 172.21.15.89:0/3238169125 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.820+0000 7f2f277fe700 1 --2- 172.21.15.89:0/3238169125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f28102800 0x7f2f2810f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.821+0000 7f2f27fff700 1 -- 172.21.15.89:0/3238169125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f28102800 msgr2=0x7f2f2810f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.821+0000 7f2f27fff700 1 --2- 172.21.15.89:0/3238169125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f28102800 0x7f2f2810f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.821+0000 7f2f27fff700 1 -- 172.21.15.89:0/3238169125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f18005040 con 0x7f2f28101e60 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.821+0000 7f2f27fff700 1 --2- 172.21.15.89:0/3238169125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f28101e60 0x7f2f2810eba0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00c830 tx=0x7f2f1c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.821+0000 7f2f24ff9700 1 -- 172.21.15.89:0/3238169125 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f1c0106f0 con 0x7f2f28101e60 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.821+0000 7f2f24ff9700 1 -- 172.21.15.89:0/3238169125 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f1c010d20 con 0x7f2f28101e60 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.821+0000 7f2f2e55d700 1 -- 172.21.15.89:0/3238169125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f28117b10 con 0x7f2f28101e60 2024-02-10T06:13:54.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.821+0000 7f2f24ff9700 1 -- 172.21.15.89:0/3238169125 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f1c0113a0 con 0x7f2f28101e60 2024-02-10T06:13:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.821+0000 7f2f2e55d700 1 -- 172.21.15.89:0/3238169125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f28118020 con 0x7f2f28101e60 2024-02-10T06:13:54.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.822+0000 7f2f2e55d700 1 -- 172.21.15.89:0/3238169125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f28104330 con 0x7f2f28101e60 2024-02-10T06:13:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.822+0000 7f2f24ff9700 1 -- 172.21.15.89:0/3238169125 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2f1c012400 con 0x7f2f28101e60 2024-02-10T06:13:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.823+0000 7f2f24ff9700 1 --2- 172.21.15.89:0/3238169125 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2f100692a0 0x7f2f1006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.823+0000 7f2f277fe700 1 --2- 172.21.15.89:0/3238169125 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2f100692a0 0x7f2f1006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.823+0000 7f2f24ff9700 1 -- 172.21.15.89:0/3238169125 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2f1c086e10 con 0x7f2f28101e60 2024-02-10T06:13:54.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.823+0000 7f2f277fe700 1 --2- 172.21.15.89:0/3238169125 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2f100692a0 0x7f2f1006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800b3b0 tx=0x7f2f18009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:54.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:54.827+0000 7f2f24ff9700 1 -- 172.21.15.89:0/3238169125 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2f1c053480 con 0x7f2f28101e60 2024-02-10T06:13:55.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.003+0000 7f2f2e55d700 1 -- 172.21.15.89:0/3238169125 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f28066750 con 0x7f2f100692a0 2024-02-10T06:13:55.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.004+0000 7f2f24ff9700 1 -- 172.21.15.89:0/3238169125 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27657 (secure 0 0 0) 0x7f2f28066750 con 0x7f2f100692a0 2024-02-10T06:13:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:13:55.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.007+0000 7f2f2e55d700 1 -- 172.21.15.89:0/3238169125 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2f100692a0 msgr2=0x7f2f1006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:55.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.007+0000 7f2f2e55d700 1 --2- 172.21.15.89:0/3238169125 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2f100692a0 0x7f2f1006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800b3b0 tx=0x7f2f18009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:55.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.007+0000 7f2f2e55d700 1 -- 172.21.15.89:0/3238169125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f28101e60 msgr2=0x7f2f2810eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:55.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.007+0000 7f2f2e55d700 1 --2- 172.21.15.89:0/3238169125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f28101e60 0x7f2f2810eba0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00c830 tx=0x7f2f1c004470 comp rx=0 tx=0).stop 2024-02-10T06:13:55.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.007+0000 7f2f2e55d700 1 -- 172.21.15.89:0/3238169125 shutdown_connections 2024-02-10T06:13:55.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.007+0000 7f2f2e55d700 1 --2- 172.21.15.89:0/3238169125 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2f100692a0 0x7f2f1006b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:55.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.007+0000 7f2f2e55d700 1 --2- 172.21.15.89:0/3238169125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f28101e60 0x7f2f2810eba0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:55.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.007+0000 7f2f2e55d700 1 --2- 172.21.15.89:0/3238169125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f28102800 0x7f2f2810f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:55.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.007+0000 7f2f2e55d700 1 -- 172.21.15.89:0/3238169125 >> 172.21.15.89:0/3238169125 conn(0x7f2f280f7a10 msgr2=0x7f2f280f9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:55.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.007+0000 7f2f2e55d700 1 -- 172.21.15.89:0/3238169125 shutdown_connections 2024-02-10T06:13:55.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:55.007+0000 7f2f2e55d700 1 -- 172.21.15.89:0/3238169125 wait complete. 2024-02-10T06:13:55.009 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-02-10T06:13:55.301 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:55 smithi089 ceph-mon[29217]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:55.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:55 smithi173 ceph-mon[36251]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:55.684 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-02-10T06:13:54.136895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001982"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-10T06:13:03.324896+0000","last_change":"2024-02-10T06:12:38.834570+0000","last_active":"2024-02-10T06:13:03.324896+0000","last_peered":"2024-02-10T06:13:03.324896+0000","last_clean":"2024-02-10T06:13:03.324896+0000","last_became_active":"2024-02-10T06:12:38.834320+0000","last_became_peered":"2024-02-10T06:12:38.834320+0000","last_unstale":"2024-02-10T06:13:03.324896+0000","last_undegraded":"2024-02-10T06:13:03.324896+0000","last_fullsized":"2024-02-10T06:13:03.324896+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T06:11:52.993144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T06:11:52.993144+0000","last_clean_scrub_stamp":"2024-02-10T06:11:52.993144+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:16:19.347522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.46500000000000002}]},{"osd":1,"last update":"Sat Feb 10 06:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Feb 10 06:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Feb 10 06:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.40000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":6.5780000000000003,"5min":6.5780000000000003,"15min":6.5780000000000003},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.59899999999999998}]},{"osd":1,"last update":"Sat Feb 10 06:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.6440000000000001,"5min":6.6440000000000001,"15min":6.6440000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.56899999999999995}]},{"osd":2,"last update":"Sat Feb 10 06:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.617,"5min":6.617,"15min":6.617},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Feb 10 06:13:40 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.437,"5min":0.437,"15min":0.437},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.30399999999999999}]},{"osd":1,"last update":"Sat Feb 10 06:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.33400000000000002}]},{"osd":3,"last update":"Sat Feb 10 06:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.63700000000000001}]},{"osd":4,"last update":"Sat Feb 10 06:13:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 06:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.312,"15min":0.30499999999999999},"min":{"1min":0.22,"5min":0.154,"15min":0.154},"max":{"1min":0.44600000000000001,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.31,"15min":0.30299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.50700000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.376}]},{"osd":2,"last update":"Sat Feb 10 06:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.434,"5min":0.434,"15min":0.434},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.40600000000000003}]},{"osd":3,"last update":"Sat Feb 10 06:13:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.75900000000000001}]},{"osd":4,"last update":"Sat Feb 10 06:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.74199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 06:13:27 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.251,"5min":0.218,"15min":0.218},"max":{"1min":0.41999999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.34000000000000002,"15min":0.33400000000000002},"min":{"1min":0.247,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.5,"5min":0.503,"15min":0.503},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Feb 10 06:12:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Feb 10 06:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":14.904,"5min":14.904,"15min":14.904},"last":0.497}]},{"osd":4,"last update":"Sat Feb 10 06:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.42899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T06:13:55.685 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T06:13:55.685 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T06:13:55.685 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T06:13:55.685 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T06:13:55.943 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:56.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:56 smithi089 ceph-mon[29217]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:13:56.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:56 smithi173 ceph-mon[36251]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T06:13:57.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 -- 172.21.15.89:0/1253927673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36180fbd50 msgr2=0x7f36180fc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:57.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 --2- 172.21.15.89:0/1253927673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36180fbd50 0x7f36180fc120 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3614004940 tx=0x7f3614009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 -- 172.21.15.89:0/1253927673 shutdown_connections 2024-02-10T06:13:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 --2- 172.21.15.89:0/1253927673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36180fc6f0 0x7f3618100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 --2- 172.21.15.89:0/1253927673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36180fbd50 0x7f36180fc120 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 -- 172.21.15.89:0/1253927673 >> 172.21.15.89:0/1253927673 conn(0x7f36180f7a10 msgr2=0x7f36180f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 -- 172.21.15.89:0/1253927673 shutdown_connections 2024-02-10T06:13:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 -- 172.21.15.89:0/1253927673 wait complete. 2024-02-10T06:13:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 Processor -- start 2024-02-10T06:13:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 -- start start 2024-02-10T06:13:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36180fbd50 0x7f361811a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:57.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36180fc6f0 0x7f361811a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361811ad30 con 0x7f36180fc6f0 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.379+0000 7f361f944700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3618113fb0 con 0x7f36180fbd50 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f361cedf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36180fc6f0 0x7f361811a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f361cedf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36180fc6f0 0x7f361811a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41638/0 (socket says 172.21.15.89:41638) 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f361cedf700 1 -- 172.21.15.89:0/3284596077 learned_addr learned my addr 172.21.15.89:0/3284596077 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f361cedf700 1 -- 172.21.15.89:0/3284596077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36180fbd50 msgr2=0x7f361811a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f361cedf700 1 --2- 172.21.15.89:0/3284596077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36180fbd50 0x7f361811a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f361cedf700 1 -- 172.21.15.89:0/3284596077 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3614005040 con 0x7f36180fc6f0 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f361cedf700 1 --2- 172.21.15.89:0/3284596077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36180fc6f0 0x7f361811a650 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f360800b790 tx=0x7f3608009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f360dffb700 1 -- 172.21.15.89:0/3284596077 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f360800f770 con 0x7f36180fc6f0 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f361f944700 1 -- 172.21.15.89:0/3284596077 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3618114230 con 0x7f36180fc6f0 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f360dffb700 1 -- 172.21.15.89:0/3284596077 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f360800fda0 con 0x7f36180fc6f0 2024-02-10T06:13:57.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f360dffb700 1 -- 172.21.15.89:0/3284596077 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f360800d310 con 0x7f36180fc6f0 2024-02-10T06:13:57.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.380+0000 7f361f944700 1 -- 172.21.15.89:0/3284596077 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3618114770 con 0x7f36180fc6f0 2024-02-10T06:13:57.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.381+0000 7f361f944700 1 -- 172.21.15.89:0/3284596077 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36180fe430 con 0x7f36180fc6f0 2024-02-10T06:13:57.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.382+0000 7f360dffb700 1 -- 172.21.15.89:0/3284596077 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3608010420 con 0x7f36180fc6f0 2024-02-10T06:13:57.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.382+0000 7f360dffb700 1 --2- 172.21.15.89:0/3284596077 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f36040692a0 0x7f360406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:13:57.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.382+0000 7f360dffb700 1 -- 172.21.15.89:0/3284596077 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3608086e70 con 0x7f36180fc6f0 2024-02-10T06:13:57.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.382+0000 7f361d6e0700 1 --2- 172.21.15.89:0/3284596077 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f36040692a0 0x7f360406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:13:57.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.382+0000 7f361d6e0700 1 --2- 172.21.15.89:0/3284596077 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f36040692a0 0x7f360406b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3614004f90 tx=0x7f3614009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:13:57.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.386+0000 7f360dffb700 1 -- 172.21.15.89:0/3284596077 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3608056f70 con 0x7f36180fc6f0 2024-02-10T06:13:57.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:57 smithi089 ceph-mon[29217]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:57.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:57 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:57.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.613+0000 7f361f944700 1 -- 172.21.15.89:0/3284596077 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f36180fb610 con 0x7f36180fc6f0 2024-02-10T06:13:57.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.613+0000 7f360dffb700 1 -- 172.21.15.89:0/3284596077 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f3608056b00 con 0x7f36180fc6f0 2024-02-10T06:13:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:13:57.615 INFO:teuthology.orchestra.run.smithi089.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T06:13:57.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.615+0000 7f361f944700 1 -- 172.21.15.89:0/3284596077 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f36040692a0 msgr2=0x7f360406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:57.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.615+0000 7f361f944700 1 --2- 172.21.15.89:0/3284596077 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f36040692a0 0x7f360406b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3614004f90 tx=0x7f3614009e30 comp rx=0 tx=0).stop 2024-02-10T06:13:57.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.615+0000 7f361f944700 1 -- 172.21.15.89:0/3284596077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36180fc6f0 msgr2=0x7f361811a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:13:57.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.615+0000 7f361f944700 1 --2- 172.21.15.89:0/3284596077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36180fc6f0 0x7f361811a650 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f360800b790 tx=0x7f3608009250 comp rx=0 tx=0).stop 2024-02-10T06:13:57.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.615+0000 7f361f944700 1 -- 172.21.15.89:0/3284596077 shutdown_connections 2024-02-10T06:13:57.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.615+0000 7f361f944700 1 --2- 172.21.15.89:0/3284596077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36180fbd50 0x7f361811a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:57.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.615+0000 7f361f944700 1 --2- 172.21.15.89:0/3284596077 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f36040692a0 0x7f360406b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:57.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.615+0000 7f361f944700 1 --2- 172.21.15.89:0/3284596077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36180fc6f0 0x7f361811a650 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:13:57.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.615+0000 7f361f944700 1 -- 172.21.15.89:0/3284596077 >> 172.21.15.89:0/3284596077 conn(0x7f36180f7a10 msgr2=0x7f361810d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:13:57.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.615+0000 7f361f944700 1 -- 172.21.15.89:0/3284596077 shutdown_connections 2024-02-10T06:13:57.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:13:57.615+0000 7f361f944700 1 -- 172.21.15.89:0/3284596077 wait complete. 2024-02-10T06:13:57.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:57 smithi173 ceph-mon[36251]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:57.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:57 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:58.266 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T06:13:58.266 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T06:13:58.266 INFO:teuthology.run_tasks:Running task print... 2024-02-10T06:13:58.275 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-02-10T06:13:58.275 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:13:58.296 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T06:13:58.296 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-02-10T06:13:58.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:58 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/3284596077' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T06:13:58.553 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:13:58.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:58 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3284596077' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T06:13:59.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:13:59 smithi089 ceph-mon[29217]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:13:59.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:13:59 smithi173 ceph-mon[36251]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:00.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.101+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/864994615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8f80fb220 msgr2=0x7fd8f80fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.101+0000 7fd8fd4a8700 1 --2- 172.21.15.89:0/864994615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8f80fb220 0x7fd8f80fb630 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8004940 tx=0x7fd8e8009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.101+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/864994615 shutdown_connections 2024-02-10T06:14:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.101+0000 7fd8fd4a8700 1 --2- 172.21.15.89:0/864994615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f80fe2b0 0x7fd8f80fe700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.101+0000 7fd8fd4a8700 1 --2- 172.21.15.89:0/864994615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8f80fb220 0x7fd8f80fb630 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.101+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/864994615 >> 172.21.15.89:0/864994615 conn(0x7fd8f80f6ca0 msgr2=0x7fd8f80f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.101+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/864994615 shutdown_connections 2024-02-10T06:14:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.101+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/864994615 wait complete. 2024-02-10T06:14:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8fd4a8700 1 Processor -- start 2024-02-10T06:14:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8fd4a8700 1 -- start start 2024-02-10T06:14:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8fd4a8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8f80fb220 0x7fd8f810da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8fd4a8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f80fe2b0 0x7fd8f810df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8fd4a8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f810e570 con 0x7fd8f80fb220 2024-02-10T06:14:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8fd4a8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f810ee10 con 0x7fd8f80fe2b0 2024-02-10T06:14:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8f6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8f80fb220 0x7fd8f810da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8f6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8f80fb220 0x7fd8f810da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55572/0 (socket says 172.21.15.89:55572) 2024-02-10T06:14:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8f6ffd700 1 -- 172.21.15.89:0/2215446260 learned_addr learned my addr 172.21.15.89:0/2215446260 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8ee7fc700 1 --2- 172.21.15.89:0/2215446260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f80fe2b0 0x7fd8f810df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8f6ffd700 1 -- 172.21.15.89:0/2215446260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f80fe2b0 msgr2=0x7fd8f810df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8f6ffd700 1 --2- 172.21.15.89:0/2215446260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f80fe2b0 0x7fd8f810df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.102+0000 7fd8f6ffd700 1 -- 172.21.15.89:0/2215446260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e8005040 con 0x7fd8f80fb220 2024-02-10T06:14:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.103+0000 7fd8f6ffd700 1 --2- 172.21.15.89:0/2215446260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8f80fb220 0x7fd8f810da30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009150 tx=0x7fd8e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.103+0000 7fd8effff700 1 -- 172.21.15.89:0/2215446260 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8e800e040 con 0x7fd8f80fb220 2024-02-10T06:14:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.103+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/2215446260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f810f090 con 0x7fd8f80fb220 2024-02-10T06:14:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.103+0000 7fd8effff700 1 -- 172.21.15.89:0/2215446260 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8e800bbf0 con 0x7fd8f80fb220 2024-02-10T06:14:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.103+0000 7fd8effff700 1 -- 172.21.15.89:0/2215446260 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8e801c630 con 0x7fd8f80fb220 2024-02-10T06:14:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.103+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/2215446260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f810f5d0 con 0x7fd8f80fb220 2024-02-10T06:14:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.104+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/2215446260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f804bed0 con 0x7fd8f80fb220 2024-02-10T06:14:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.104+0000 7fd8effff700 1 -- 172.21.15.89:0/2215446260 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd8e800bd60 con 0x7fd8f80fb220 2024-02-10T06:14:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.104+0000 7fd8effff700 1 --2- 172.21.15.89:0/2215446260 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd8e40692a0 0x7fd8e406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:00.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.105+0000 7fd8ee7fc700 1 --2- 172.21.15.89:0/2215446260 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd8e40692a0 0x7fd8e406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:00.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.105+0000 7fd8effff700 1 -- 172.21.15.89:0/2215446260 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd8e80881c0 con 0x7fd8f80fb220 2024-02-10T06:14:00.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.105+0000 7fd8ee7fc700 1 --2- 172.21.15.89:0/2215446260 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd8e40692a0 0x7fd8e406b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0004d60 tx=0x7fd8e000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:00.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.110+0000 7fd8effff700 1 -- 172.21.15.89:0/2215446260 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd8e80582c0 con 0x7fd8f80fb220 2024-02-10T06:14:00.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.294+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/2215446260 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fd8f8105a40 con 0x7fd8f80fb220 2024-02-10T06:14:00.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.300+0000 7fd8effff700 1 -- 172.21.15.89:0/2215446260 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fd8e8057e50 con 0x7fd8f80fb220 2024-02-10T06:14:00.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.302+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/2215446260 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd8e40692a0 msgr2=0x7fd8e406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:00.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.302+0000 7fd8fd4a8700 1 --2- 172.21.15.89:0/2215446260 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd8e40692a0 0x7fd8e406b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0004d60 tx=0x7fd8e000b040 comp rx=0 tx=0).stop 2024-02-10T06:14:00.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.302+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/2215446260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8f80fb220 msgr2=0x7fd8f810da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:00.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.302+0000 7fd8fd4a8700 1 --2- 172.21.15.89:0/2215446260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8f80fb220 0x7fd8f810da30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009150 tx=0x7fd8e8009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:00.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.302+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/2215446260 shutdown_connections 2024-02-10T06:14:00.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.302+0000 7fd8fd4a8700 1 --2- 172.21.15.89:0/2215446260 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd8e40692a0 0x7fd8e406b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:00.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.302+0000 7fd8fd4a8700 1 --2- 172.21.15.89:0/2215446260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8f80fb220 0x7fd8f810da30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:00.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.302+0000 7fd8fd4a8700 1 --2- 172.21.15.89:0/2215446260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f80fe2b0 0x7fd8f810df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:00.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.302+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/2215446260 >> 172.21.15.89:0/2215446260 conn(0x7fd8f80f6ca0 msgr2=0x7fd8f81014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:00.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.302+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/2215446260 shutdown_connections 2024-02-10T06:14:00.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:00.302+0000 7fd8fd4a8700 1 -- 172.21.15.89:0/2215446260 wait complete. 2024-02-10T06:14:01.011 INFO:teuthology.run_tasks:Running task print... 2024-02-10T06:14:01.021 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-02-10T06:14:01.021 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:14:01.036 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T06:14:01.036 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph orch status' 2024-02-10T06:14:01.290 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:01.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:01 smithi089 ceph-mon[29217]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:01.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:01 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2215446260' entity='client.admin' 2024-02-10T06:14:01.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:01.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:01.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:01.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:01 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:01.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:01 smithi173 ceph-mon[36251]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:01.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:01 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2215446260' entity='client.admin' 2024-02-10T06:14:01.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:01.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:01.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:01.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:01 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:02.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.986+0000 7fc4ccfae700 1 -- 172.21.15.89:0/1904982761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4c80fcef0 msgr2=0x7fc4c80fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:02.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.986+0000 7fc4ccfae700 1 --2- 172.21.15.89:0/1904982761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4c80fcef0 0x7fc4c80fd320 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009ce0 tx=0x7fc4c0004d10 comp rx=0 tx=0).stop 2024-02-10T06:14:02.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.986+0000 7fc4ccfae700 1 -- 172.21.15.89:0/1904982761 shutdown_connections 2024-02-10T06:14:02.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.986+0000 7fc4ccfae700 1 --2- 172.21.15.89:0/1904982761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c80fe0e0 0x7fc4c80fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:02.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.986+0000 7fc4ccfae700 1 --2- 172.21.15.89:0/1904982761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4c80fcef0 0x7fc4c80fd320 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:02.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.987+0000 7fc4ccfae700 1 -- 172.21.15.89:0/1904982761 >> 172.21.15.89:0/1904982761 conn(0x7fc4c80f8620 msgr2=0x7fc4c80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:02.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.987+0000 7fc4ccfae700 1 -- 172.21.15.89:0/1904982761 shutdown_connections 2024-02-10T06:14:02.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.987+0000 7fc4ccfae700 1 -- 172.21.15.89:0/1904982761 wait complete. 2024-02-10T06:14:02.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.987+0000 7fc4ccfae700 1 Processor -- start 2024-02-10T06:14:02.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.987+0000 7fc4ccfae700 1 -- start start 2024-02-10T06:14:02.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.987+0000 7fc4ccfae700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c80fcef0 0x7fc4c8071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.987+0000 7fc4ccfae700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4c80fe0e0 0x7fc4c80702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.987+0000 7fc4ccfae700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80708e0 con 0x7fc4c80fe0e0 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.987+0000 7fc4ccfae700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c8070a50 con 0x7fc4c80fcef0 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.987+0000 7fc4c5d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4c80fe0e0 0x7fc4c80702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.987+0000 7fc4c5d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4c80fe0e0 0x7fc4c80702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55592/0 (socket says 172.21.15.89:55592) 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.988+0000 7fc4c5d9b700 1 -- 172.21.15.89:0/3510218819 learned_addr learned my addr 172.21.15.89:0/3510218819 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.988+0000 7fc4c659c700 1 --2- 172.21.15.89:0/3510218819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c80fcef0 0x7fc4c8071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.988+0000 7fc4c5d9b700 1 -- 172.21.15.89:0/3510218819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c80fcef0 msgr2=0x7fc4c8071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.988+0000 7fc4c5d9b700 1 --2- 172.21.15.89:0/3510218819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c80fcef0 0x7fc4c8071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.988+0000 7fc4c5d9b700 1 -- 172.21.15.89:0/3510218819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c0009940 con 0x7fc4c80fe0e0 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.988+0000 7fc4c5d9b700 1 --2- 172.21.15.89:0/3510218819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4c80fe0e0 0x7fc4c80702e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000c830 tx=0x7fc4b0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:02.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.988+0000 7fc4beffd700 1 -- 172.21.15.89:0/3510218819 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4b00106f0 con 0x7fc4c80fe0e0 2024-02-10T06:14:02.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.988+0000 7fc4ccfae700 1 -- 172.21.15.89:0/3510218819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c8070d30 con 0x7fc4c80fe0e0 2024-02-10T06:14:02.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.988+0000 7fc4ccfae700 1 -- 172.21.15.89:0/3510218819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c8115b10 con 0x7fc4c80fe0e0 2024-02-10T06:14:02.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.988+0000 7fc4beffd700 1 -- 172.21.15.89:0/3510218819 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4b0010d20 con 0x7fc4c80fe0e0 2024-02-10T06:14:02.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.988+0000 7fc4beffd700 1 -- 172.21.15.89:0/3510218819 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4b00113a0 con 0x7fc4c80fe0e0 2024-02-10T06:14:02.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.989+0000 7fc4ccfae700 1 -- 172.21.15.89:0/3510218819 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c804bed0 con 0x7fc4c80fe0e0 2024-02-10T06:14:02.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.990+0000 7fc4beffd700 1 -- 172.21.15.89:0/3510218819 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc4b0010850 con 0x7fc4c80fe0e0 2024-02-10T06:14:02.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.990+0000 7fc4beffd700 1 --2- 172.21.15.89:0/3510218819 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc4b40692a0 0x7fc4b406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:02.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.990+0000 7fc4beffd700 1 -- 172.21.15.89:0/3510218819 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc4b0086d90 con 0x7fc4c80fe0e0 2024-02-10T06:14:02.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.990+0000 7fc4c659c700 1 --2- 172.21.15.89:0/3510218819 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc4b40692a0 0x7fc4b406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:02.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.991+0000 7fc4c659c700 1 --2- 172.21.15.89:0/3510218819 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc4b40692a0 0x7fc4b406b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8071bf0 tx=0x7fc4c0004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:02.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:02.995+0000 7fc4beffd700 1 -- 172.21.15.89:0/3510218819 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc4b0056f30 con 0x7fc4c80fe0e0 2024-02-10T06:14:03.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.206+0000 7fc4ccfae700 1 -- 172.21.15.89:0/3510218819 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4c8102ac0 con 0x7fc4b40692a0 2024-02-10T06:14:03.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.207+0000 7fc4beffd700 1 -- 172.21.15.89:0/3510218819 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc4c8102ac0 con 0x7fc4b40692a0 2024-02-10T06:14:03.208 INFO:teuthology.orchestra.run.smithi089.stdout:Backend: cephadm 2024-02-10T06:14:03.208 INFO:teuthology.orchestra.run.smithi089.stdout:Available: Yes 2024-02-10T06:14:03.208 INFO:teuthology.orchestra.run.smithi089.stdout:Paused: No 2024-02-10T06:14:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.209+0000 7fc4ccfae700 1 -- 172.21.15.89:0/3510218819 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc4b40692a0 msgr2=0x7fc4b406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.209+0000 7fc4ccfae700 1 --2- 172.21.15.89:0/3510218819 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc4b40692a0 0x7fc4b406b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8071bf0 tx=0x7fc4c0004d10 comp rx=0 tx=0).stop 2024-02-10T06:14:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.209+0000 7fc4ccfae700 1 -- 172.21.15.89:0/3510218819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4c80fe0e0 msgr2=0x7fc4c80702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.209+0000 7fc4ccfae700 1 --2- 172.21.15.89:0/3510218819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4c80fe0e0 0x7fc4c80702e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000c830 tx=0x7fc4b0004470 comp rx=0 tx=0).stop 2024-02-10T06:14:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.209+0000 7fc4ccfae700 1 -- 172.21.15.89:0/3510218819 shutdown_connections 2024-02-10T06:14:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.209+0000 7fc4ccfae700 1 --2- 172.21.15.89:0/3510218819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c80fcef0 0x7fc4c8071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.209+0000 7fc4ccfae700 1 --2- 172.21.15.89:0/3510218819 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc4b40692a0 0x7fc4b406b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.209+0000 7fc4ccfae700 1 --2- 172.21.15.89:0/3510218819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4c80fe0e0 0x7fc4c80702e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.209+0000 7fc4ccfae700 1 -- 172.21.15.89:0/3510218819 >> 172.21.15.89:0/3510218819 conn(0x7fc4c80f8620 msgr2=0x7fc4c81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.209+0000 7fc4ccfae700 1 -- 172.21.15.89:0/3510218819 shutdown_connections 2024-02-10T06:14:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:03.209+0000 7fc4ccfae700 1 -- 172.21.15.89:0/3510218819 wait complete. 2024-02-10T06:14:03.525 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:03 smithi089 ceph-mon[29217]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:03.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:03 smithi173 ceph-mon[36251]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:03.929 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-10T06:14:04.183 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:04.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:04 smithi089 ceph-mon[29217]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:04.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:04 smithi173 ceph-mon[36251]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:05.527 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:05 smithi089 ceph-mon[29217]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:05.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:05 smithi173 ceph-mon[36251]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:05.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.743+0000 7f5c77d54700 1 -- 172.21.15.89:0/2323614214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c700fe0e0 msgr2=0x7f5c700fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:05.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.743+0000 7f5c77d54700 1 --2- 172.21.15.89:0/2323614214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c700fe0e0 0x7f5c700fe550 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c004940 tx=0x7f5c6c009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.743+0000 7f5c77d54700 1 -- 172.21.15.89:0/2323614214 shutdown_connections 2024-02-10T06:14:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.743+0000 7f5c77d54700 1 --2- 172.21.15.89:0/2323614214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c700fe0e0 0x7f5c700fe550 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.743+0000 7f5c77d54700 1 --2- 172.21.15.89:0/2323614214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c700fcef0 0x7f5c700fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.743+0000 7f5c77d54700 1 -- 172.21.15.89:0/2323614214 >> 172.21.15.89:0/2323614214 conn(0x7f5c700f8620 msgr2=0x7f5c700faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.743+0000 7f5c77d54700 1 -- 172.21.15.89:0/2323614214 shutdown_connections 2024-02-10T06:14:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.743+0000 7f5c77d54700 1 -- 172.21.15.89:0/2323614214 wait complete. 2024-02-10T06:14:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c77d54700 1 Processor -- start 2024-02-10T06:14:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c77d54700 1 -- start start 2024-02-10T06:14:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c77d54700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c700fcef0 0x7f5c70071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c77d54700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c700fe0e0 0x7f5c700702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c77d54700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700708e0 con 0x7f5c700fcef0 2024-02-10T06:14:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c77d54700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70070a50 con 0x7f5c700fe0e0 2024-02-10T06:14:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c75af0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c700fcef0 0x7f5c70071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c75af0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c700fcef0 0x7f5c70071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55610/0 (socket says 172.21.15.89:55610) 2024-02-10T06:14:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c75af0700 1 -- 172.21.15.89:0/3304750789 learned_addr learned my addr 172.21.15.89:0/3304750789 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c752ef700 1 --2- 172.21.15.89:0/3304750789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c700fe0e0 0x7f5c700702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c75af0700 1 -- 172.21.15.89:0/3304750789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c700fe0e0 msgr2=0x7f5c700702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c75af0700 1 --2- 172.21.15.89:0/3304750789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c700fe0e0 0x7f5c700702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c75af0700 1 -- 172.21.15.89:0/3304750789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c005040 con 0x7f5c700fcef0 2024-02-10T06:14:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c75af0700 1 --2- 172.21.15.89:0/3304750789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c700fcef0 0x7f5c70071e60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000c800 tx=0x7f5c60004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.744+0000 7f5c667fc700 1 -- 172.21.15.89:0/3304750789 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c600106f0 con 0x7f5c700fcef0 2024-02-10T06:14:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.745+0000 7f5c667fc700 1 -- 172.21.15.89:0/3304750789 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c60010d20 con 0x7f5c700fcef0 2024-02-10T06:14:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.745+0000 7f5c77d54700 1 -- 172.21.15.89:0/3304750789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c70070d30 con 0x7f5c700fcef0 2024-02-10T06:14:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.745+0000 7f5c667fc700 1 -- 172.21.15.89:0/3304750789 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c600113a0 con 0x7f5c700fcef0 2024-02-10T06:14:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.745+0000 7f5c77d54700 1 -- 172.21.15.89:0/3304750789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c701159c0 con 0x7f5c700fcef0 2024-02-10T06:14:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.745+0000 7f5c77d54700 1 -- 172.21.15.89:0/3304750789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7004bed0 con 0x7f5c700fcef0 2024-02-10T06:14:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.746+0000 7f5c667fc700 1 -- 172.21.15.89:0/3304750789 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5c60012400 con 0x7f5c700fcef0 2024-02-10T06:14:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.746+0000 7f5c667fc700 1 --2- 172.21.15.89:0/3304750789 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f5c5c0692b0 0x7f5c5c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.746+0000 7f5c667fc700 1 -- 172.21.15.89:0/3304750789 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5c60086d10 con 0x7f5c700fcef0 2024-02-10T06:14:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.746+0000 7f5c752ef700 1 --2- 172.21.15.89:0/3304750789 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f5c5c0692b0 0x7f5c5c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.747+0000 7f5c752ef700 1 --2- 172.21.15.89:0/3304750789 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f5c5c0692b0 0x7f5c5c06b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00b7a0 tx=0x7f5c6c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:05.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.751+0000 7f5c667fc700 1 -- 172.21.15.89:0/3304750789 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5c60056eb0 con 0x7f5c700fcef0 2024-02-10T06:14:05.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.942+0000 7f5c77d54700 1 -- 172.21.15.89:0/3304750789 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c70102ac0 con 0x7f5c5c0692b0 2024-02-10T06:14:05.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.950+0000 7f5c667fc700 1 -- 172.21.15.89:0/3304750789 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7f5c70102ac0 con 0x7f5c5c0692b0 2024-02-10T06:14:05.951 INFO:teuthology.orchestra.run.smithi089.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:14:05.951 INFO:teuthology.orchestra.run.smithi089.stdout:alertmanager.smithi089 smithi089 *:9093,9094 running (3m) 2m ago 4m 25.6M - 0.25.0 c8568f914cd2 667ad25cb367 2024-02-10T06:14:05.951 INFO:teuthology.orchestra.run.smithi089.stdout:crash.smithi089 smithi089 running (5m) 2m ago 5m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 e3aac8b18020 2024-02-10T06:14:05.951 INFO:teuthology.orchestra.run.smithi089.stdout:crash.smithi173 smithi173 running (4m) 64s ago 4m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 269616f4b479 2024-02-10T06:14:05.951 INFO:teuthology.orchestra.run.smithi089.stdout:grafana.smithi089 smithi089 *:3000 running (3m) 2m ago 4m 79.8M - 9.4.7 954c08fa6188 a6f72269ee53 2024-02-10T06:14:05.951 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.smithi089.pwmrye smithi089 *:9283 running (6m) 2m ago 6m 472M - 17.2.7-505-g72ce7c1d a2a0bd126c96 1cd3521b2ebb 2024-02-10T06:14:05.951 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.smithi173.swdirm smithi173 *:8443,9283 running (4m) 64s ago 4m 432M - 17.2.7-505-g72ce7c1d a2a0bd126c96 6430108d8d2e 2024-02-10T06:14:05.951 INFO:teuthology.orchestra.run.smithi089.stdout:mon.smithi089 smithi089 running (6m) 2m ago 6m 42.0M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 0109d6119759 2024-02-10T06:14:05.952 INFO:teuthology.orchestra.run.smithi089.stdout:mon.smithi173 smithi173 running (3m) 64s ago 3m 38.0M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 08af8aefe37b 2024-02-10T06:14:05.952 INFO:teuthology.orchestra.run.smithi089.stdout:node-exporter.smithi089 smithi089 *:9100 running (5m) 2m ago 5m 15.2M - 1.5.0 0da6a335fe13 18b1f6a49022 2024-02-10T06:14:05.952 INFO:teuthology.orchestra.run.smithi089.stdout:node-exporter.smithi173 smithi173 *:9100 running (4m) 64s ago 4m 16.2M - 1.5.0 0da6a335fe13 f7d0b906813f 2024-02-10T06:14:05.952 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0 smithi089 running (3m) 2m ago 3m 36.3M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 e372a21351db 2024-02-10T06:14:05.952 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1 smithi089 running (2m) 2m ago 2m 36.0M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 42f862e2f6ca 2024-02-10T06:14:05.952 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2 smithi089 running (2m) 2m ago 2m 36.7M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 69b16e6f89fb 2024-02-10T06:14:05.952 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3 smithi173 running (118s) 64s ago 118s 38.3M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 997c2081faec 2024-02-10T06:14:05.952 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4 smithi173 running (93s) 64s ago 93s 37.2M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 d3c6d0e77356 2024-02-10T06:14:05.952 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5 smithi173 running (69s) 64s ago 68s 10.3M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 bb845c14ca98 2024-02-10T06:14:05.952 INFO:teuthology.orchestra.run.smithi089.stdout:prometheus.smithi089 smithi089 *:9095 running (3m) 2m ago 4m 38.2M - 2.43.0 a07b618ecd1d 04539fe88109 2024-02-10T06:14:05.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.952+0000 7f5c77d54700 1 -- 172.21.15.89:0/3304750789 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f5c5c0692b0 msgr2=0x7f5c5c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:05.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.952+0000 7f5c77d54700 1 --2- 172.21.15.89:0/3304750789 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f5c5c0692b0 0x7f5c5c06b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00b7a0 tx=0x7f5c6c009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:05.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.952+0000 7f5c77d54700 1 -- 172.21.15.89:0/3304750789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c700fcef0 msgr2=0x7f5c70071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:05.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.952+0000 7f5c77d54700 1 --2- 172.21.15.89:0/3304750789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c700fcef0 0x7f5c70071e60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000c800 tx=0x7f5c60004470 comp rx=0 tx=0).stop 2024-02-10T06:14:05.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.952+0000 7f5c77d54700 1 -- 172.21.15.89:0/3304750789 shutdown_connections 2024-02-10T06:14:05.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.952+0000 7f5c77d54700 1 --2- 172.21.15.89:0/3304750789 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f5c5c0692b0 0x7f5c5c06b760 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:05.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.952+0000 7f5c77d54700 1 --2- 172.21.15.89:0/3304750789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c700fcef0 0x7f5c70071e60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:05.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.952+0000 7f5c77d54700 1 --2- 172.21.15.89:0/3304750789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c700fe0e0 0x7f5c700702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:05.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.952+0000 7f5c77d54700 1 -- 172.21.15.89:0/3304750789 >> 172.21.15.89:0/3304750789 conn(0x7f5c700f8620 msgr2=0x7f5c701013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:05.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.952+0000 7f5c77d54700 1 -- 172.21.15.89:0/3304750789 shutdown_connections 2024-02-10T06:14:05.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:05.952+0000 7f5c77d54700 1 -- 172.21.15.89:0/3304750789 wait complete. 2024-02-10T06:14:06.591 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-10T06:14:06.848 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:07.596 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:07 smithi089 ceph-mon[29217]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:07.596 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:07 smithi089 ceph-mon[29217]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:07.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:07 smithi173 ceph-mon[36251]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:07.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:07 smithi173 ceph-mon[36251]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.424+0000 7f18ef7c2700 1 -- 172.21.15.89:0/3184374665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18e80fbe40 msgr2=0x7f18e80fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.424+0000 7f18ef7c2700 1 --2- 172.21.15.89:0/3184374665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18e80fbe40 0x7f18e80fc250 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f18d8004940 tx=0x7f18d8009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.424+0000 7f18ef7c2700 1 -- 172.21.15.89:0/3184374665 shutdown_connections 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.424+0000 7f18ef7c2700 1 --2- 172.21.15.89:0/3184374665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e80fcfb0 0x7f18e80fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.424+0000 7f18ef7c2700 1 --2- 172.21.15.89:0/3184374665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18e80fbe40 0x7f18e80fc250 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.424+0000 7f18ef7c2700 1 -- 172.21.15.89:0/3184374665 >> 172.21.15.89:0/3184374665 conn(0x7f18e80f7520 msgr2=0x7f18e80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.424+0000 7f18ef7c2700 1 -- 172.21.15.89:0/3184374665 shutdown_connections 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.424+0000 7f18ef7c2700 1 -- 172.21.15.89:0/3184374665 wait complete. 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ef7c2700 1 Processor -- start 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ef7c2700 1 -- start start 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ef7c2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18e80fbe40 0x7f18e80fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ef7c2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e80fcfb0 0x7f18e80fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ef7c2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e80fe980 con 0x7f18e80fbe40 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ef7c2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e80feaf0 con 0x7f18e80fcfb0 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ed55e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18e80fbe40 0x7f18e80fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ed55e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18e80fbe40 0x7f18e80fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55628/0 (socket says 172.21.15.89:55628) 2024-02-10T06:14:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ed55e700 1 -- 172.21.15.89:0/388636089 learned_addr learned my addr 172.21.15.89:0/388636089 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ecd5d700 1 --2- 172.21.15.89:0/388636089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e80fcfb0 0x7f18e80fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ed55e700 1 -- 172.21.15.89:0/388636089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e80fcfb0 msgr2=0x7f18e80fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ed55e700 1 --2- 172.21.15.89:0/388636089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e80fcfb0 0x7f18e80fe380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ed55e700 1 -- 172.21.15.89:0/388636089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d8005040 con 0x7f18e80fbe40 2024-02-10T06:14:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ed55e700 1 --2- 172.21.15.89:0/388636089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18e80fbe40 0x7f18e80fff00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f18d80093b0 tx=0x7f18d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.425+0000 7f18ddffb700 1 -- 172.21.15.89:0/388636089 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18d800e040 con 0x7f18e80fbe40 2024-02-10T06:14:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.426+0000 7f18ddffb700 1 -- 172.21.15.89:0/388636089 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18d800bcc0 con 0x7f18e80fbe40 2024-02-10T06:14:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.426+0000 7f18ef7c2700 1 -- 172.21.15.89:0/388636089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e80fed70 con 0x7f18e80fbe40 2024-02-10T06:14:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.426+0000 7f18ddffb700 1 -- 172.21.15.89:0/388636089 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18d801c630 con 0x7f18e80fbe40 2024-02-10T06:14:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.426+0000 7f18ef7c2700 1 -- 172.21.15.89:0/388636089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e8115960 con 0x7f18e80fbe40 2024-02-10T06:14:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.427+0000 7f18ef7c2700 1 -- 172.21.15.89:0/388636089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e804bed0 con 0x7f18e80fbe40 2024-02-10T06:14:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.427+0000 7f18ddffb700 1 -- 172.21.15.89:0/388636089 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f18d80139f0 con 0x7f18e80fbe40 2024-02-10T06:14:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.427+0000 7f18ddffb700 1 --2- 172.21.15.89:0/388636089 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18d40692a0 0x7f18d406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.428+0000 7f18ddffb700 1 -- 172.21.15.89:0/388636089 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f18d80882a0 con 0x7f18e80fbe40 2024-02-10T06:14:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.428+0000 7f18ecd5d700 1 --2- 172.21.15.89:0/388636089 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18d40692a0 0x7f18d406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.428+0000 7f18ecd5d700 1 --2- 172.21.15.89:0/388636089 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18d40692a0 0x7f18d406b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f18e4007fd0 tx=0x7f18e4009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.432+0000 7f18ddffb700 1 -- 172.21.15.89:0/388636089 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f18d8058450 con 0x7f18e80fbe40 2024-02-10T06:14:08.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.622+0000 7f18ef7c2700 1 -- 172.21.15.89:0/388636089 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f18e8066660 con 0x7f18d40692a0 2024-02-10T06:14:08.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.626+0000 7f18ddffb700 1 -- 172.21.15.89:0/388636089 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7f18e8066660 con 0x7f18d40692a0 2024-02-10T06:14:08.627 INFO:teuthology.orchestra.run.smithi089.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-10T06:14:08.627 INFO:teuthology.orchestra.run.smithi089.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-02-10T06:14:08.627 INFO:teuthology.orchestra.run.smithi089.stdout:crash 2/2 2m ago 5m * 2024-02-10T06:14:08.627 INFO:teuthology.orchestra.run.smithi089.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-02-10T06:14:08.627 INFO:teuthology.orchestra.run.smithi089.stdout:mgr 2/2 2m ago 5m count:2 2024-02-10T06:14:08.627 INFO:teuthology.orchestra.run.smithi089.stdout:mon 2/2 2m ago 4m smithi089:172.21.15.89=smithi089;smithi173:172.21.15.173=smithi173;count:2 2024-02-10T06:14:08.628 INFO:teuthology.orchestra.run.smithi089.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-02-10T06:14:08.628 INFO:teuthology.orchestra.run.smithi089.stdout:osd 6 2m ago - 2024-02-10T06:14:08.628 INFO:teuthology.orchestra.run.smithi089.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-02-10T06:14:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.628+0000 7f18ef7c2700 1 -- 172.21.15.89:0/388636089 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18d40692a0 msgr2=0x7f18d406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.628+0000 7f18ef7c2700 1 --2- 172.21.15.89:0/388636089 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18d40692a0 0x7f18d406b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f18e4007fd0 tx=0x7f18e4009040 comp rx=0 tx=0).stop 2024-02-10T06:14:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.628+0000 7f18ef7c2700 1 -- 172.21.15.89:0/388636089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18e80fbe40 msgr2=0x7f18e80fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.628+0000 7f18ef7c2700 1 --2- 172.21.15.89:0/388636089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18e80fbe40 0x7f18e80fff00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f18d80093b0 tx=0x7f18d8009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.629+0000 7f18ef7c2700 1 -- 172.21.15.89:0/388636089 shutdown_connections 2024-02-10T06:14:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.629+0000 7f18ef7c2700 1 --2- 172.21.15.89:0/388636089 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18d40692a0 0x7f18d406b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.629+0000 7f18ef7c2700 1 --2- 172.21.15.89:0/388636089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18e80fbe40 0x7f18e80fff00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.629+0000 7f18ef7c2700 1 --2- 172.21.15.89:0/388636089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e80fcfb0 0x7f18e80fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.629+0000 7f18ef7c2700 1 -- 172.21.15.89:0/388636089 >> 172.21.15.89:0/388636089 conn(0x7f18e80f7520 msgr2=0x7f18e8065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.629+0000 7f18ef7c2700 1 -- 172.21.15.89:0/388636089 shutdown_connections 2024-02-10T06:14:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:08.629+0000 7f18ef7c2700 1 -- 172.21.15.89:0/388636089 wait complete. 2024-02-10T06:14:09.262 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:09 smithi089 ceph-mon[29217]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:09.263 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:09 smithi089 ceph-mon[29217]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:09.292 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-10T06:14:09.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:09 smithi173 ceph-mon[36251]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:09.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:09 smithi173 ceph-mon[36251]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:11.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.114+0000 7fda6851e700 1 -- 172.21.15.89:0/657380907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda600fcf30 msgr2=0x7fda600fd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:11.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.114+0000 7fda6851e700 1 --2- 172.21.15.89:0/657380907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda600fcf30 0x7fda600fd340 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fda5c004940 tx=0x7fda5c009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:11.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.114+0000 7fda6851e700 1 -- 172.21.15.89:0/657380907 shutdown_connections 2024-02-10T06:14:11.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.114+0000 7fda6851e700 1 --2- 172.21.15.89:0/657380907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda600fe1e0 0x7fda600fe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:11.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.114+0000 7fda6851e700 1 --2- 172.21.15.89:0/657380907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda600fcf30 0x7fda600fd340 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:11.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.114+0000 7fda6851e700 1 -- 172.21.15.89:0/657380907 >> 172.21.15.89:0/657380907 conn(0x7fda600f86b0 msgr2=0x7fda600faac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:11.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.114+0000 7fda6851e700 1 -- 172.21.15.89:0/657380907 shutdown_connections 2024-02-10T06:14:11.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.114+0000 7fda6851e700 1 -- 172.21.15.89:0/657380907 wait complete. 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.115+0000 7fda6851e700 1 Processor -- start 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.115+0000 7fda6851e700 1 -- start start 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.115+0000 7fda6851e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda600fcf30 0x7fda60071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.115+0000 7fda6851e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda600fe1e0 0x7fda600702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.115+0000 7fda6851e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda600708e0 con 0x7fda600fe1e0 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.115+0000 7fda6851e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda60070a50 con 0x7fda600fcf30 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.115+0000 7fda65ab9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda600fe1e0 0x7fda600702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.115+0000 7fda65ab9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda600fe1e0 0x7fda600702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59464/0 (socket says 172.21.15.89:59464) 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.115+0000 7fda65ab9700 1 -- 172.21.15.89:0/2610758129 learned_addr learned my addr 172.21.15.89:0/2610758129 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.115+0000 7fda662ba700 1 --2- 172.21.15.89:0/2610758129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda600fcf30 0x7fda60071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.116+0000 7fda65ab9700 1 -- 172.21.15.89:0/2610758129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda600fcf30 msgr2=0x7fda60071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.116+0000 7fda65ab9700 1 --2- 172.21.15.89:0/2610758129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda600fcf30 0x7fda60071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.116+0000 7fda65ab9700 1 -- 172.21.15.89:0/2610758129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c005040 con 0x7fda600fe1e0 2024-02-10T06:14:11.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.116+0000 7fda65ab9700 1 --2- 172.21.15.89:0/2610758129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda600fe1e0 0x7fda600702e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c8c0 tx=0x7fda500044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:11.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.116+0000 7fda56ffd700 1 -- 172.21.15.89:0/2610758129 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda5000a760 con 0x7fda600fe1e0 2024-02-10T06:14:11.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.116+0000 7fda6851e700 1 -- 172.21.15.89:0/2610758129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda60070d30 con 0x7fda600fe1e0 2024-02-10T06:14:11.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.116+0000 7fda56ffd700 1 -- 172.21.15.89:0/2610758129 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda5000ad90 con 0x7fda600fe1e0 2024-02-10T06:14:11.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.116+0000 7fda56ffd700 1 -- 172.21.15.89:0/2610758129 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda500113a0 con 0x7fda600fe1e0 2024-02-10T06:14:11.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.116+0000 7fda6851e700 1 -- 172.21.15.89:0/2610758129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda60115990 con 0x7fda600fe1e0 2024-02-10T06:14:11.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.117+0000 7fda6851e700 1 -- 172.21.15.89:0/2610758129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6004bed0 con 0x7fda600fe1e0 2024-02-10T06:14:11.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.117+0000 7fda56ffd700 1 -- 172.21.15.89:0/2610758129 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fda50012400 con 0x7fda600fe1e0 2024-02-10T06:14:11.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.118+0000 7fda56ffd700 1 --2- 172.21.15.89:0/2610758129 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fda4c0692a0 0x7fda4c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:11.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.118+0000 7fda56ffd700 1 -- 172.21.15.89:0/2610758129 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fda50086e60 con 0x7fda600fe1e0 2024-02-10T06:14:11.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.118+0000 7fda662ba700 1 --2- 172.21.15.89:0/2610758129 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fda4c0692a0 0x7fda4c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:11.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.118+0000 7fda662ba700 1 --2- 172.21.15.89:0/2610758129 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fda4c0692a0 0x7fda4c06b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00b840 tx=0x7fda5c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:11.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.122+0000 7fda56ffd700 1 -- 172.21.15.89:0/2610758129 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fda50056f60 con 0x7fda600fe1e0 2024-02-10T06:14:11.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.312+0000 7fda6851e700 1 -- 172.21.15.89:0/2610758129 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fda60102b30 con 0x7fda4c0692a0 2024-02-10T06:14:11.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.313+0000 7fda56ffd700 1 -- 172.21.15.89:0/2610758129 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fda60102b30 con 0x7fda4c0692a0 2024-02-10T06:14:11.314 INFO:teuthology.orchestra.run.smithi089.stdout:HOST ADDR LABELS STATUS 2024-02-10T06:14:11.314 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089 172.21.15.89 2024-02-10T06:14:11.314 INFO:teuthology.orchestra.run.smithi089.stdout:smithi173 172.21.15.173 2024-02-10T06:14:11.314 INFO:teuthology.orchestra.run.smithi089.stdout:2 hosts in cluster 2024-02-10T06:14:11.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.315+0000 7fda6851e700 1 -- 172.21.15.89:0/2610758129 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fda4c0692a0 msgr2=0x7fda4c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:11.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.315+0000 7fda6851e700 1 --2- 172.21.15.89:0/2610758129 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fda4c0692a0 0x7fda4c06b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00b840 tx=0x7fda5c009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:11.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.315+0000 7fda6851e700 1 -- 172.21.15.89:0/2610758129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda600fe1e0 msgr2=0x7fda600702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:11.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.315+0000 7fda6851e700 1 --2- 172.21.15.89:0/2610758129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda600fe1e0 0x7fda600702e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c8c0 tx=0x7fda500044c0 comp rx=0 tx=0).stop 2024-02-10T06:14:11.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.315+0000 7fda6851e700 1 -- 172.21.15.89:0/2610758129 shutdown_connections 2024-02-10T06:14:11.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.315+0000 7fda6851e700 1 --2- 172.21.15.89:0/2610758129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda600fcf30 0x7fda60071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:11.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.315+0000 7fda6851e700 1 --2- 172.21.15.89:0/2610758129 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fda4c0692a0 0x7fda4c06b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:11.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.315+0000 7fda6851e700 1 --2- 172.21.15.89:0/2610758129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda600fe1e0 0x7fda600702e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:11.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.315+0000 7fda6851e700 1 -- 172.21.15.89:0/2610758129 >> 172.21.15.89:0/2610758129 conn(0x7fda600f86b0 msgr2=0x7fda60101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:11.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.315+0000 7fda6851e700 1 -- 172.21.15.89:0/2610758129 shutdown_connections 2024-02-10T06:14:11.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:11.315+0000 7fda6851e700 1 -- 172.21.15.89:0/2610758129 wait complete. 2024-02-10T06:14:11.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:11 smithi089 ceph-mon[29217]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:11.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:11 smithi173 ceph-mon[36251]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:11.909 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-10T06:14:12.162 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:12.487 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:12 smithi089 ceph-mon[29217]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:12.487 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:12 smithi173 ceph-mon[36251]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:13.499 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:13 smithi089 ceph-mon[29217]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:13 smithi173 ceph-mon[36251]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:13.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.717+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/1443555433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a380fda80 msgr2=0x7f1a380ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:13.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.717+0000 7f1a3d9b5700 1 --2- 172.21.15.89:0/1443555433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a380fda80 0x7f1a380ffe60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1a28004940 tx=0x7f1a28009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:13.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.717+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/1443555433 shutdown_connections 2024-02-10T06:14:13.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.717+0000 7f1a3d9b5700 1 --2- 172.21.15.89:0/1443555433 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a381003a0 0x7f1a38102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:13.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.717+0000 7f1a3d9b5700 1 --2- 172.21.15.89:0/1443555433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a380fda80 0x7f1a380ffe60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:13.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.717+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/1443555433 >> 172.21.15.89:0/1443555433 conn(0x7f1a380f74e0 msgr2=0x7f1a380f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:13.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.717+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/1443555433 shutdown_connections 2024-02-10T06:14:13.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.717+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/1443555433 wait complete. 2024-02-10T06:14:13.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.717+0000 7f1a3d9b5700 1 Processor -- start 2024-02-10T06:14:13.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.717+0000 7f1a3d9b5700 1 -- start start 2024-02-10T06:14:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a3d9b5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a380fda80 0x7f1a3810d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a3d9b5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a381003a0 0x7f1a3810de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a3d9b5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3810e4e0 con 0x7f1a380fda80 2024-02-10T06:14:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a3d9b5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3810ed20 con 0x7f1a381003a0 2024-02-10T06:14:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a36ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a380fda80 0x7f1a3810d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a36ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a380fda80 0x7f1a3810d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59484/0 (socket says 172.21.15.89:59484) 2024-02-10T06:14:13.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a36ffd700 1 -- 172.21.15.89:0/553852798 learned_addr learned my addr 172.21.15.89:0/553852798 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a36ffd700 1 -- 172.21.15.89:0/553852798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a381003a0 msgr2=0x7f1a3810de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a36ffd700 1 --2- 172.21.15.89:0/553852798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a381003a0 0x7f1a3810de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a36ffd700 1 -- 172.21.15.89:0/553852798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a28005040 con 0x7f1a380fda80 2024-02-10T06:14:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a36ffd700 1 --2- 172.21.15.89:0/553852798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a380fda80 0x7f1a3810d910 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1a280093b0 tx=0x7f1a28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a2ffff700 1 -- 172.21.15.89:0/553852798 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a2800e040 con 0x7f1a380fda80 2024-02-10T06:14:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/553852798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3810efa0 con 0x7f1a380fda80 2024-02-10T06:14:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a2ffff700 1 -- 172.21.15.89:0/553852798 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a2800bcc0 con 0x7f1a380fda80 2024-02-10T06:14:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a2ffff700 1 -- 172.21.15.89:0/553852798 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a2801c630 con 0x7f1a380fda80 2024-02-10T06:14:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.718+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/553852798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3810f4e0 con 0x7f1a380fda80 2024-02-10T06:14:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.719+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/553852798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a38107ad0 con 0x7f1a380fda80 2024-02-10T06:14:13.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.720+0000 7f1a2ffff700 1 -- 172.21.15.89:0/553852798 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1a280139f0 con 0x7f1a380fda80 2024-02-10T06:14:13.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.720+0000 7f1a2ffff700 1 --2- 172.21.15.89:0/553852798 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1a240692a0 0x7f1a2406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:13.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.720+0000 7f1a2ffff700 1 -- 172.21.15.89:0/553852798 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1a28088250 con 0x7f1a380fda80 2024-02-10T06:14:13.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.720+0000 7f1a2e7fc700 1 --2- 172.21.15.89:0/553852798 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1a240692a0 0x7f1a2406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:13.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.721+0000 7f1a2e7fc700 1 --2- 172.21.15.89:0/553852798 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1a240692a0 0x7f1a2406b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1a20006fd0 tx=0x7f1a20006db0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:13.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.725+0000 7f1a2ffff700 1 -- 172.21.15.89:0/553852798 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1a28058350 con 0x7f1a380fda80 2024-02-10T06:14:13.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.914+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/553852798 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a3810f880 con 0x7f1a240692a0 2024-02-10T06:14:13.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.915+0000 7f1a2ffff700 1 -- 172.21.15.89:0/553852798 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f1a3810f880 con 0x7f1a240692a0 2024-02-10T06:14:13.916 INFO:teuthology.orchestra.run.smithi089.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-10T06:14:13.916 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040126400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-02-10T06:14:13.916 INFO:teuthology.orchestra.run.smithi089.stdout:smithi173 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204019S400BGN 372G No 69s ago Has a FileSystem, LVM detected 2024-02-10T06:14:13.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.917+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/553852798 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1a240692a0 msgr2=0x7f1a2406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.917+0000 7f1a3d9b5700 1 --2- 172.21.15.89:0/553852798 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1a240692a0 0x7f1a2406b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1a20006fd0 tx=0x7f1a20006db0 comp rx=0 tx=0).stop 2024-02-10T06:14:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.917+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/553852798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a380fda80 msgr2=0x7f1a3810d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.917+0000 7f1a3d9b5700 1 --2- 172.21.15.89:0/553852798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a380fda80 0x7f1a3810d910 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1a280093b0 tx=0x7f1a28009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.918+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/553852798 shutdown_connections 2024-02-10T06:14:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.918+0000 7f1a3d9b5700 1 --2- 172.21.15.89:0/553852798 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1a240692a0 0x7f1a2406b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.918+0000 7f1a3d9b5700 1 --2- 172.21.15.89:0/553852798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a380fda80 0x7f1a3810d910 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.918+0000 7f1a3d9b5700 1 --2- 172.21.15.89:0/553852798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a381003a0 0x7f1a3810de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.918+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/553852798 >> 172.21.15.89:0/553852798 conn(0x7f1a380f74e0 msgr2=0x7f1a380f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.918+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/553852798 shutdown_connections 2024-02-10T06:14:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:13.918+0000 7f1a3d9b5700 1 -- 172.21.15.89:0/553852798 wait complete. 2024-02-10T06:14:14.609 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:14:14.619 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T06:14:14.619 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-02-10T06:14:14.878 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:15.284 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:15 smithi089 ceph-mon[29217]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:15.284 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:15 smithi089 ceph-mon[29217]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:15.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:15 smithi173 ceph-mon[36251]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:15.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:15 smithi173 ceph-mon[36251]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:16.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.358+0000 7f991a9a2700 1 -- 172.21.15.89:0/2627151151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99140fdfa0 msgr2=0x7f99140fe3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.358+0000 7f991a9a2700 1 --2- 172.21.15.89:0/2627151151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99140fdfa0 0x7f99140fe3f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9904004940 tx=0x7f9904009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.358+0000 7f991a9a2700 1 -- 172.21.15.89:0/2627151151 shutdown_connections 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.358+0000 7f991a9a2700 1 --2- 172.21.15.89:0/2627151151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99140fdfa0 0x7f99140fe3f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.358+0000 7f991a9a2700 1 --2- 172.21.15.89:0/2627151151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99140fcd90 0x7f99140fd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.358+0000 7f991a9a2700 1 -- 172.21.15.89:0/2627151151 >> 172.21.15.89:0/2627151151 conn(0x7f99140f8450 msgr2=0x7f99140fa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.358+0000 7f991a9a2700 1 -- 172.21.15.89:0/2627151151 shutdown_connections 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.358+0000 7f991a9a2700 1 -- 172.21.15.89:0/2627151151 wait complete. 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991a9a2700 1 Processor -- start 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991a9a2700 1 -- start start 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991a9a2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99140fcd90 0x7f991410d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991a9a2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99140fdfa0 0x7f991410dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:16.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991a9a2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991410e290 con 0x7f99140fdfa0 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991a9a2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991410eb30 con 0x7f99140fcd90 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991919f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99140fdfa0 0x7f991410dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991919f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99140fdfa0 0x7f991410dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59508/0 (socket says 172.21.15.89:59508) 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991919f700 1 -- 172.21.15.89:0/2996162205 learned_addr learned my addr 172.21.15.89:0/2996162205 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f99199a0700 1 --2- 172.21.15.89:0/2996162205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99140fcd90 0x7f991410d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991919f700 1 -- 172.21.15.89:0/2996162205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99140fcd90 msgr2=0x7f991410d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991919f700 1 --2- 172.21.15.89:0/2996162205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99140fcd90 0x7f991410d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991919f700 1 -- 172.21.15.89:0/2996162205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9904005040 con 0x7f99140fdfa0 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f991919f700 1 --2- 172.21.15.89:0/2996162205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99140fdfa0 0x7f991410dc90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9904009fd0 tx=0x7f9904009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f990a7fc700 1 -- 172.21.15.89:0/2996162205 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f990400e040 con 0x7f99140fdfa0 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f990a7fc700 1 -- 172.21.15.89:0/2996162205 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f990400bcc0 con 0x7f99140fdfa0 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.359+0000 7f990a7fc700 1 -- 172.21.15.89:0/2996162205 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f990401c630 con 0x7f99140fdfa0 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.360+0000 7f991a9a2700 1 -- 172.21.15.89:0/2996162205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991410edb0 con 0x7f99140fdfa0 2024-02-10T06:14:16.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.360+0000 7f991a9a2700 1 -- 172.21.15.89:0/2996162205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991410f2a0 con 0x7f99140fdfa0 2024-02-10T06:14:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.361+0000 7f991a9a2700 1 -- 172.21.15.89:0/2996162205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991404bed0 con 0x7f99140fdfa0 2024-02-10T06:14:16.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.361+0000 7f990a7fc700 1 -- 172.21.15.89:0/2996162205 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9904013980 con 0x7f99140fdfa0 2024-02-10T06:14:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.361+0000 7f990a7fc700 1 --2- 172.21.15.89:0/2996162205 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99000692b0 0x7f990006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.361+0000 7f990a7fc700 1 -- 172.21.15.89:0/2996162205 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9904088310 con 0x7f99140fdfa0 2024-02-10T06:14:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.361+0000 7f99199a0700 1 --2- 172.21.15.89:0/2996162205 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99000692b0 0x7f990006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:16.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.362+0000 7f99199a0700 1 --2- 172.21.15.89:0/2996162205 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99000692b0 0x7f990006b760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9910007fd0 tx=0x7f9910009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:16.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.366+0000 7f990a7fc700 1 -- 172.21.15.89:0/2996162205 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9904058410 con 0x7f99140fdfa0 2024-02-10T06:14:16.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:16.576+0000 7f991a9a2700 1 -- 172.21.15.89:0/2996162205 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f991410f7d0 con 0x7f99000692b0 2024-02-10T06:14:17.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:17 smithi089 ceph-mon[29217]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:17.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:17 smithi089 ceph-mon[29217]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:17.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:17 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-10T06:14:17.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:17 smithi173 ceph-mon[36251]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:17.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:17 smithi173 ceph-mon[36251]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:14:17.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:17 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-10T06:14:18.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.256+0000 7f990a7fc700 1 -- 172.21.15.89:0/2996162205 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f991410f7d0 con 0x7f99000692b0 2024-02-10T06:14:18.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.259+0000 7f991a9a2700 1 -- 172.21.15.89:0/2996162205 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99000692b0 msgr2=0x7f990006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:18.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.259+0000 7f991a9a2700 1 --2- 172.21.15.89:0/2996162205 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99000692b0 0x7f990006b760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9910007fd0 tx=0x7f9910009040 comp rx=0 tx=0).stop 2024-02-10T06:14:18.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.259+0000 7f991a9a2700 1 -- 172.21.15.89:0/2996162205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99140fdfa0 msgr2=0x7f991410dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:18.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.259+0000 7f991a9a2700 1 --2- 172.21.15.89:0/2996162205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99140fdfa0 0x7f991410dc90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9904009fd0 tx=0x7f9904009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:18.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.259+0000 7f991a9a2700 1 -- 172.21.15.89:0/2996162205 shutdown_connections 2024-02-10T06:14:18.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.259+0000 7f991a9a2700 1 --2- 172.21.15.89:0/2996162205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99140fcd90 0x7f991410d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:18.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.259+0000 7f991a9a2700 1 --2- 172.21.15.89:0/2996162205 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99000692b0 0x7f990006b760 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:18.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.259+0000 7f991a9a2700 1 --2- 172.21.15.89:0/2996162205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99140fdfa0 0x7f991410dc90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:18.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.259+0000 7f991a9a2700 1 -- 172.21.15.89:0/2996162205 >> 172.21.15.89:0/2996162205 conn(0x7f99140f8450 msgr2=0x7f99141011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:18.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.260+0000 7f991a9a2700 1 -- 172.21.15.89:0/2996162205 shutdown_connections 2024-02-10T06:14:18.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:18.260+0000 7f991a9a2700 1 -- 172.21.15.89:0/2996162205 wait complete. 2024-02-10T06:14:18.399 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:18 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-10T06:14:18.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:18 smithi173 ceph-mon[36251]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T06:14:18.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:18 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-10T06:14:18.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:18 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-10T06:14:18.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:18 smithi089 ceph-mon[29217]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T06:14:18.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:18 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-10T06:14:18.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:14:18.235+0000 7f0dfeba4700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:14:18.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-10T06:14:19.532 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: pgmap v168: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:19.533 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-10T06:14:19.533 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T06:14:19.533 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-10T06:14:19.533 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:14:19.533 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-10T06:14:19.533 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: osdmap e44: 6 total, 6 up, 6 in 2024-02-10T06:14:19.534 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-10T06:14:19.534 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: fsmap cephfs:0 2024-02-10T06:14:19.534 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: Saving service mds.cephfs spec with placement count:4 2024-02-10T06:14:19.534 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:19.534 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:19.534 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:19.535 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:19.535 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:19.535 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.mgkvmj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:14:19.535 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.mgkvmj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:14:19.535 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:19.535 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:19 smithi089 ceph-mon[29217]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:14:19.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: pgmap v168: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:19.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-10T06:14:19.660 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T06:14:19.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-10T06:14:19.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T06:14:19.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-10T06:14:19.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: osdmap e44: 6 total, 6 up, 6 in 2024-02-10T06:14:19.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-10T06:14:19.661 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: fsmap cephfs:0 2024-02-10T06:14:19.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: Saving service mds.cephfs spec with placement count:4 2024-02-10T06:14:19.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:19.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:19.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:19.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:19.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:19.662 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.mgkvmj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:14:19.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.mgkvmj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:14:19.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:19.663 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:19 smithi173 ceph-mon[36251]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:14:20.428 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:20.510 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:20 smithi089 ceph-mon[29217]: Deploying daemon mds.cephfs.smithi089.mgkvmj on smithi089 2024-02-10T06:14:20.510 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:20 smithi089 ceph-mon[29217]: osdmap e45: 6 total, 6 up, 6 in 2024-02-10T06:14:20.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:20 smithi173 ceph-mon[36251]: Deploying daemon mds.cephfs.smithi089.mgkvmj on smithi089 2024-02-10T06:14:20.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:20 smithi173 ceph-mon[36251]: osdmap e45: 6 total, 6 up, 6 in 2024-02-10T06:14:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:21 smithi089 ceph-mon[29217]: pgmap v172: 65 pgs: 41 active+clean, 9 creating+peering, 15 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:21 smithi089 ceph-mon[29217]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T06:14:21.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:21 smithi089 ceph-mon[29217]: osdmap e46: 6 total, 6 up, 6 in 2024-02-10T06:14:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:21 smithi173 ceph-mon[36251]: pgmap v172: 65 pgs: 41 active+clean, 9 creating+peering, 15 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:21 smithi173 ceph-mon[36251]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T06:14:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:21 smithi173 ceph-mon[36251]: osdmap e46: 6 total, 6 up, 6 in 2024-02-10T06:14:22.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.550+0000 7f92fdc1c700 1 -- 172.21.15.89:0/4067543985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92f80fcf30 msgr2=0x7f92f80fd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:22.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.550+0000 7f92fdc1c700 1 --2- 172.21.15.89:0/4067543985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92f80fcf30 0x7f92f80fd340 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f92e8004940 tx=0x7f92e8009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:22.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.550+0000 7f92fdc1c700 1 -- 172.21.15.89:0/4067543985 shutdown_connections 2024-02-10T06:14:22.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.550+0000 7f92fdc1c700 1 --2- 172.21.15.89:0/4067543985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f80fe1e0 0x7f92f80fe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:22.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.550+0000 7f92fdc1c700 1 --2- 172.21.15.89:0/4067543985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92f80fcf30 0x7f92f80fd340 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:22.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.550+0000 7f92fdc1c700 1 -- 172.21.15.89:0/4067543985 >> 172.21.15.89:0/4067543985 conn(0x7f92f80f86b0 msgr2=0x7f92f80faac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:22.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.550+0000 7f92fdc1c700 1 -- 172.21.15.89:0/4067543985 shutdown_connections 2024-02-10T06:14:22.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.550+0000 7f92fdc1c700 1 -- 172.21.15.89:0/4067543985 wait complete. 2024-02-10T06:14:22.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.550+0000 7f92fdc1c700 1 Processor -- start 2024-02-10T06:14:22.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.550+0000 7f92fdc1c700 1 -- start start 2024-02-10T06:14:22.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92fdc1c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f80fcf30 0x7f92f810d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92fdc1c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92f80fe1e0 0x7f92f810ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92fdc1c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f810e3d0 con 0x7f92f80fe1e0 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92fdc1c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f810ec70 con 0x7f92f80fcf30 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92f6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92f80fe1e0 0x7f92f810ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92f6ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92f80fe1e0 0x7f92f810ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59076/0 (socket says 172.21.15.89:59076) 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92f6ffd700 1 -- 172.21.15.89:0/2756406559 learned_addr learned my addr 172.21.15.89:0/2756406559 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92f77fe700 1 --2- 172.21.15.89:0/2756406559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f80fcf30 0x7f92f810d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92f6ffd700 1 -- 172.21.15.89:0/2756406559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f80fcf30 msgr2=0x7f92f810d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92f6ffd700 1 --2- 172.21.15.89:0/2756406559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f80fcf30 0x7f92f810d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92f6ffd700 1 -- 172.21.15.89:0/2756406559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e8005040 con 0x7f92f80fe1e0 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92f6ffd700 1 --2- 172.21.15.89:0/2756406559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92f80fe1e0 0x7f92f810ddd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00eb40 tx=0x7f92ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92dffff700 1 -- 172.21.15.89:0/2756406559 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ec007cc0 con 0x7f92f80fe1e0 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92dffff700 1 -- 172.21.15.89:0/2756406559 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92ec00f040 con 0x7f92f80fe1e0 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92fdc1c700 1 -- 172.21.15.89:0/2756406559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f810eef0 con 0x7f92f80fe1e0 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92dffff700 1 -- 172.21.15.89:0/2756406559 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ec00ca10 con 0x7f92f80fe1e0 2024-02-10T06:14:22.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.551+0000 7f92fdc1c700 1 -- 172.21.15.89:0/2756406559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f810f430 con 0x7f92f80fe1e0 2024-02-10T06:14:22.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.552+0000 7f92fdc1c700 1 -- 172.21.15.89:0/2756406559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f804bed0 con 0x7f92f80fe1e0 2024-02-10T06:14:22.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.553+0000 7f92dffff700 1 -- 172.21.15.89:0/2756406559 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f92ec00cc80 con 0x7f92f80fe1e0 2024-02-10T06:14:22.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.553+0000 7f92dffff700 1 --2- 172.21.15.89:0/2756406559 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92e00692a0 0x7f92e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:22.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.553+0000 7f92dffff700 1 -- 172.21.15.89:0/2756406559 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f92ec086d80 con 0x7f92f80fe1e0 2024-02-10T06:14:22.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.553+0000 7f92f77fe700 1 --2- 172.21.15.89:0/2756406559 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92e00692a0 0x7f92e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:22.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.554+0000 7f92f77fe700 1 --2- 172.21.15.89:0/2756406559 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92e00692a0 0x7f92e006b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009b30 tx=0x7f92e8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:22.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.558+0000 7f92dffff700 1 -- 172.21.15.89:0/2756406559 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f92ec04dbd0 con 0x7f92f80fe1e0 2024-02-10T06:14:22.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.785+0000 7f92fdc1c700 1 -- 172.21.15.89:0/2756406559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f92f8105a20 con 0x7f92f80fe1e0 2024-02-10T06:14:22.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.785+0000 7f92dffff700 1 -- 172.21.15.89:0/2756406559 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f92ec056b70 con 0x7f92f80fe1e0 2024-02-10T06:14:22.786 INFO:teuthology.orchestra.run.smithi089.stdout:e2 2024-02-10T06:14:22.786 INFO:teuthology.orchestra.run.smithi089.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:14:22.786 INFO:teuthology.orchestra.run.smithi089.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:14:22.787 INFO:teuthology.orchestra.run.smithi089.stdout:legacy client fscid: 1 2024-02-10T06:14:22.787 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:14:22.787 INFO:teuthology.orchestra.run.smithi089.stdout:Filesystem 'cephfs' (1) 2024-02-10T06:14:22.787 INFO:teuthology.orchestra.run.smithi089.stdout:fs_name cephfs 2024-02-10T06:14:22.787 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 2 2024-02-10T06:14:22.787 INFO:teuthology.orchestra.run.smithi089.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-10T06:14:22.787 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-02-10T06:14:18.236041+0000 2024-02-10T06:14:22.787 INFO:teuthology.orchestra.run.smithi089.stdout:modified 2024-02-10T06:14:18.236094+0000 2024-02-10T06:14:22.787 INFO:teuthology.orchestra.run.smithi089.stdout:tableserver 0 2024-02-10T06:14:22.788 INFO:teuthology.orchestra.run.smithi089.stdout:root 0 2024-02-10T06:14:22.788 INFO:teuthology.orchestra.run.smithi089.stdout:session_timeout 60 2024-02-10T06:14:22.788 INFO:teuthology.orchestra.run.smithi089.stdout:session_autoclose 300 2024-02-10T06:14:22.788 INFO:teuthology.orchestra.run.smithi089.stdout:max_file_size 1099511627776 2024-02-10T06:14:22.788 INFO:teuthology.orchestra.run.smithi089.stdout:required_client_features {} 2024-02-10T06:14:22.788 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure 0 2024-02-10T06:14:22.788 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure_osd_epoch 0 2024-02-10T06:14:22.788 INFO:teuthology.orchestra.run.smithi089.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:14:22.788 INFO:teuthology.orchestra.run.smithi089.stdout:max_mds 1 2024-02-10T06:14:22.788 INFO:teuthology.orchestra.run.smithi089.stdout:in 2024-02-10T06:14:22.788 INFO:teuthology.orchestra.run.smithi089.stdout:up {} 2024-02-10T06:14:22.789 INFO:teuthology.orchestra.run.smithi089.stdout:failed 2024-02-10T06:14:22.789 INFO:teuthology.orchestra.run.smithi089.stdout:damaged 2024-02-10T06:14:22.789 INFO:teuthology.orchestra.run.smithi089.stdout:stopped 2024-02-10T06:14:22.789 INFO:teuthology.orchestra.run.smithi089.stdout:data_pools [3] 2024-02-10T06:14:22.789 INFO:teuthology.orchestra.run.smithi089.stdout:metadata_pool 2 2024-02-10T06:14:22.789 INFO:teuthology.orchestra.run.smithi089.stdout:inline_data disabled 2024-02-10T06:14:22.789 INFO:teuthology.orchestra.run.smithi089.stdout:balancer 2024-02-10T06:14:22.789 INFO:teuthology.orchestra.run.smithi089.stdout:standby_count_wanted 0 2024-02-10T06:14:22.789 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:14:22.789 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:14:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.787+0000 7f92fdc1c700 1 -- 172.21.15.89:0/2756406559 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92e00692a0 msgr2=0x7f92e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.787+0000 7f92fdc1c700 1 --2- 172.21.15.89:0/2756406559 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92e00692a0 0x7f92e006b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009b30 tx=0x7f92e8009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.787+0000 7f92fdc1c700 1 -- 172.21.15.89:0/2756406559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92f80fe1e0 msgr2=0x7f92f810ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:22.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.787+0000 7f92fdc1c700 1 --2- 172.21.15.89:0/2756406559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92f80fe1e0 0x7f92f810ddd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00eb40 tx=0x7f92ec00b040 comp rx=0 tx=0).stop 2024-02-10T06:14:22.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.787+0000 7f92fdc1c700 1 -- 172.21.15.89:0/2756406559 shutdown_connections 2024-02-10T06:14:22.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.787+0000 7f92fdc1c700 1 --2- 172.21.15.89:0/2756406559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f80fcf30 0x7f92f810d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:22.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.787+0000 7f92fdc1c700 1 --2- 172.21.15.89:0/2756406559 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92e00692a0 0x7f92e006b750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:22.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.787+0000 7f92fdc1c700 1 --2- 172.21.15.89:0/2756406559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92f80fe1e0 0x7f92f810ddd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:22.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.787+0000 7f92fdc1c700 1 -- 172.21.15.89:0/2756406559 >> 172.21.15.89:0/2756406559 conn(0x7f92f80f86b0 msgr2=0x7f92f8101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:22.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.787+0000 7f92fdc1c700 1 -- 172.21.15.89:0/2756406559 shutdown_connections 2024-02-10T06:14:22.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:22.787+0000 7f92fdc1c700 1 -- 172.21.15.89:0/2756406559 wait complete. 2024-02-10T06:14:22.791 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 2 2024-02-10T06:14:23.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: pgmap v174: 65 pgs: 50 active+clean, 9 creating+peering, 6 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:23.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:23.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:23.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:23.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi173.tumnkz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:14:23.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi173.tumnkz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:14:23.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:23.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: Deploying daemon mds.cephfs.smithi173.tumnkz on smithi173 2024-02-10T06:14:23.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2756406559' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:14:23.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: daemon mds.cephfs.smithi089.mgkvmj assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-10T06:14:23.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T06:14:23.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-10T06:14:23.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: Cluster is now healthy 2024-02-10T06:14:23.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.89:6826/3380248129,v1:172.21.15.89:6827/3380248129] up:boot 2024-02-10T06:14:23.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:creating} 2024-02-10T06:14:23.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi089.mgkvmj"}]: dispatch 2024-02-10T06:14:23.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:23 smithi173 ceph-mon[36251]: daemon mds.cephfs.smithi089.mgkvmj is now active in filesystem cephfs as rank 0 2024-02-10T06:14:23.522 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:14:23.532 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T06:14:23.533 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-02-10T06:14:23.787 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:23.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: pgmap v174: 65 pgs: 50 active+clean, 9 creating+peering, 6 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi173.tumnkz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi173.tumnkz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: Deploying daemon mds.cephfs.smithi173.tumnkz on smithi173 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2756406559' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: daemon mds.cephfs.smithi089.mgkvmj assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-10T06:14:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: Cluster is now healthy 2024-02-10T06:14:23.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.89:6826/3380248129,v1:172.21.15.89:6827/3380248129] up:boot 2024-02-10T06:14:23.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:creating} 2024-02-10T06:14:23.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi089.mgkvmj"}]: dispatch 2024-02-10T06:14:23.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:23 smithi089 ceph-mon[29217]: daemon mds.cephfs.smithi089.mgkvmj is now active in filesystem cephfs as rank 0 2024-02-10T06:14:25.169 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:25 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.89:6826/3380248129,v1:172.21.15.89:6827/3380248129] up:active 2024-02-10T06:14:25.170 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:25 smithi089 ceph-mon[29217]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 2024-02-10T06:14:25.170 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:25 smithi089 ceph-mon[29217]: pgmap v175: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 519 B/s wr, 2 op/s 2024-02-10T06:14:25.170 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:25 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:25.170 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:25 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:25.170 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:25 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:25.170 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:25 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.evydlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:14:25.170 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:25 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.evydlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:14:25.170 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:25 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:25.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:25 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.89:6826/3380248129,v1:172.21.15.89:6827/3380248129] up:active 2024-02-10T06:14:25.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:25 smithi173 ceph-mon[36251]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 2024-02-10T06:14:25.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:25 smithi173 ceph-mon[36251]: pgmap v175: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 519 B/s wr, 2 op/s 2024-02-10T06:14:25.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:25 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:25.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:25 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:25.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:25 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:25.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:25 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.evydlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:14:25.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:25 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.evydlq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:14:25.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:25 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.426+0000 7fd651d9f700 1 -- 172.21.15.89:0/3288612020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd64c0fbe40 msgr2=0x7fd64c0fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.426+0000 7fd651d9f700 1 --2- 172.21.15.89:0/3288612020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd64c0fbe40 0x7fd64c0fc250 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd634004940 tx=0x7fd634009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.426+0000 7fd651d9f700 1 -- 172.21.15.89:0/3288612020 shutdown_connections 2024-02-10T06:14:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.426+0000 7fd651d9f700 1 --2- 172.21.15.89:0/3288612020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd64c0fcfb0 0x7fd64c0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.426+0000 7fd651d9f700 1 --2- 172.21.15.89:0/3288612020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd64c0fbe40 0x7fd64c0fc250 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.426+0000 7fd651d9f700 1 -- 172.21.15.89:0/3288612020 >> 172.21.15.89:0/3288612020 conn(0x7fd64c0f74e0 msgr2=0x7fd64c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.426+0000 7fd651d9f700 1 -- 172.21.15.89:0/3288612020 shutdown_connections 2024-02-10T06:14:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.426+0000 7fd651d9f700 1 -- 172.21.15.89:0/3288612020 wait complete. 2024-02-10T06:14:25.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.427+0000 7fd651d9f700 1 Processor -- start 2024-02-10T06:14:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.427+0000 7fd651d9f700 1 -- start start 2024-02-10T06:14:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.427+0000 7fd651d9f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd64c0fbe40 0x7fd64c110650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.428+0000 7fd651d9f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd64c0fcfb0 0x7fd64c112ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.428+0000 7fd651d9f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c066510 con 0x7fd64c0fcfb0 2024-02-10T06:14:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.428+0000 7fd651d9f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c066650 con 0x7fd64c0fbe40 2024-02-10T06:14:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.428+0000 7fd64affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd64c0fcfb0 0x7fd64c112ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:25.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.428+0000 7fd64affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd64c0fcfb0 0x7fd64c112ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59098/0 (socket says 172.21.15.89:59098) 2024-02-10T06:14:25.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.428+0000 7fd64affd700 1 -- 172.21.15.89:0/1936421392 learned_addr learned my addr 172.21.15.89:0/1936421392 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:25.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.428+0000 7fd64b7fe700 1 --2- 172.21.15.89:0/1936421392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd64c0fbe40 0x7fd64c110650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:25.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.428+0000 7fd64affd700 1 -- 172.21.15.89:0/1936421392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd64c0fbe40 msgr2=0x7fd64c110650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:25.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.428+0000 7fd64affd700 1 --2- 172.21.15.89:0/1936421392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd64c0fbe40 0x7fd64c110650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:25.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.428+0000 7fd64affd700 1 -- 172.21.15.89:0/1936421392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd634005040 con 0x7fd64c0fcfb0 2024-02-10T06:14:25.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.429+0000 7fd64affd700 1 --2- 172.21.15.89:0/1936421392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd64c0fcfb0 0x7fd64c112ba0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00b7c0 tx=0x7fd63c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:25.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.429+0000 7fd643fff700 1 -- 172.21.15.89:0/1936421392 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd63c00f6f0 con 0x7fd64c0fcfb0 2024-02-10T06:14:25.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.429+0000 7fd643fff700 1 -- 172.21.15.89:0/1936421392 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd63c00fd20 con 0x7fd64c0fcfb0 2024-02-10T06:14:25.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.429+0000 7fd643fff700 1 -- 172.21.15.89:0/1936421392 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd63c0103a0 con 0x7fd64c0fcfb0 2024-02-10T06:14:25.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.429+0000 7fd651d9f700 1 -- 172.21.15.89:0/1936421392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64c113250 con 0x7fd64c0fcfb0 2024-02-10T06:14:25.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.429+0000 7fd651d9f700 1 -- 172.21.15.89:0/1936421392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64c1136e0 con 0x7fd64c0fcfb0 2024-02-10T06:14:25.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.430+0000 7fd651d9f700 1 -- 172.21.15.89:0/1936421392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64c04bed0 con 0x7fd64c0fcfb0 2024-02-10T06:14:25.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.430+0000 7fd643fff700 1 -- 172.21.15.89:0/1936421392 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd63c011400 con 0x7fd64c0fcfb0 2024-02-10T06:14:25.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.431+0000 7fd643fff700 1 --2- 172.21.15.89:0/1936421392 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd6380692b0 0x7fd63806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:25.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.431+0000 7fd643fff700 1 -- 172.21.15.89:0/1936421392 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd63c086f70 con 0x7fd64c0fcfb0 2024-02-10T06:14:25.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.431+0000 7fd64b7fe700 1 --2- 172.21.15.89:0/1936421392 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd6380692b0 0x7fd63806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:25.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.431+0000 7fd64b7fe700 1 --2- 172.21.15.89:0/1936421392 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd6380692b0 0x7fd63806b760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd63400b840 tx=0x7fd634009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:25.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.436+0000 7fd643fff700 1 -- 172.21.15.89:0/1936421392 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd63c056d30 con 0x7fd64c0fcfb0 2024-02-10T06:14:25.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:25.670+0000 7fd651d9f700 1 -- 172.21.15.89:0/1936421392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fd64c1139b0 con 0x7fd64c0fcfb0 2024-02-10T06:14:26.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.025+0000 7fd643fff700 1 -- 172.21.15.89:0/1936421392 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v7) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd63c0568c0 con 0x7fd64c0fcfb0 2024-02-10T06:14:26.026 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:26 smithi089 ceph-mon[29217]: Deploying daemon mds.cephfs.smithi089.evydlq on smithi089 2024-02-10T06:14:26.026 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:26 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] up:boot 2024-02-10T06:14:26.026 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:26 smithi089 ceph-mon[29217]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 1 up:standby 2024-02-10T06:14:26.026 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:26 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi173.tumnkz"}]: dispatch 2024-02-10T06:14:26.026 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:26 smithi089 ceph-mon[29217]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 1 up:standby 2024-02-10T06:14:26.026 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:26 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1936421392' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-10T06:14:26.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.027+0000 7fd651d9f700 1 -- 172.21.15.89:0/1936421392 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd6380692b0 msgr2=0x7fd63806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:26.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.027+0000 7fd651d9f700 1 --2- 172.21.15.89:0/1936421392 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd6380692b0 0x7fd63806b760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd63400b840 tx=0x7fd634009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:26.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.027+0000 7fd651d9f700 1 -- 172.21.15.89:0/1936421392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd64c0fcfb0 msgr2=0x7fd64c112ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:26.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.027+0000 7fd651d9f700 1 --2- 172.21.15.89:0/1936421392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd64c0fcfb0 0x7fd64c112ba0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00b7c0 tx=0x7fd63c009250 comp rx=0 tx=0).stop 2024-02-10T06:14:26.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.027+0000 7fd651d9f700 1 -- 172.21.15.89:0/1936421392 shutdown_connections 2024-02-10T06:14:26.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.027+0000 7fd651d9f700 1 --2- 172.21.15.89:0/1936421392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd64c0fbe40 0x7fd64c110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:26.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.027+0000 7fd651d9f700 1 --2- 172.21.15.89:0/1936421392 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fd6380692b0 0x7fd63806b760 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:26.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.027+0000 7fd651d9f700 1 --2- 172.21.15.89:0/1936421392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd64c0fcfb0 0x7fd64c112ba0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:26.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.027+0000 7fd651d9f700 1 -- 172.21.15.89:0/1936421392 >> 172.21.15.89:0/1936421392 conn(0x7fd64c0f74e0 msgr2=0x7fd64c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:26.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.027+0000 7fd651d9f700 1 -- 172.21.15.89:0/1936421392 shutdown_connections 2024-02-10T06:14:26.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:26.027+0000 7fd651d9f700 1 -- 172.21.15.89:0/1936421392 wait complete. 2024-02-10T06:14:26.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:26 smithi173 ceph-mon[36251]: Deploying daemon mds.cephfs.smithi089.evydlq on smithi089 2024-02-10T06:14:26.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:26 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] up:boot 2024-02-10T06:14:26.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:26 smithi173 ceph-mon[36251]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 1 up:standby 2024-02-10T06:14:26.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:26 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi173.tumnkz"}]: dispatch 2024-02-10T06:14:26.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:26 smithi173 ceph-mon[36251]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 1 up:standby 2024-02-10T06:14:26.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:26 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/1936421392' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-10T06:14:26.674 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:14:26.685 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T06:14:26.685 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-02-10T06:14:27.080 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:27 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/1936421392' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-10T06:14:27.080 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:27 smithi173 ceph-mon[36251]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 1 up:standby 2024-02-10T06:14:27.080 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:27 smithi173 ceph-mon[36251]: pgmap v176: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 777 B/s wr, 2 op/s 2024-02-10T06:14:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:27.290 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:27 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1936421392' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-10T06:14:27.290 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:27 smithi089 ceph-mon[29217]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 1 up:standby 2024-02-10T06:14:27.290 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:27 smithi089 ceph-mon[29217]: pgmap v176: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 777 B/s wr, 2 op/s 2024-02-10T06:14:28.324 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:28 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:28.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:28 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:28.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.804+0000 7f1fde2c3700 1 -- 172.21.15.89:0/910935976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80fcef0 msgr2=0x7f1fd80fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.804+0000 7f1fde2c3700 1 --2- 172.21.15.89:0/910935976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80fcef0 0x7f1fd80fd320 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8004940 tx=0x7f1fc8009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.804+0000 7f1fde2c3700 1 -- 172.21.15.89:0/910935976 shutdown_connections 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.804+0000 7f1fde2c3700 1 --2- 172.21.15.89:0/910935976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fd80fe0e0 0x7f1fd80fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.804+0000 7f1fde2c3700 1 --2- 172.21.15.89:0/910935976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80fcef0 0x7f1fd80fd320 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.804+0000 7f1fde2c3700 1 -- 172.21.15.89:0/910935976 >> 172.21.15.89:0/910935976 conn(0x7f1fd80f8620 msgr2=0x7f1fd80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.804+0000 7f1fde2c3700 1 -- 172.21.15.89:0/910935976 shutdown_connections 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.804+0000 7f1fde2c3700 1 -- 172.21.15.89:0/910935976 wait complete. 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.804+0000 7f1fde2c3700 1 Processor -- start 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fde2c3700 1 -- start start 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fde2c3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fd80fcef0 0x7f1fd810d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fde2c3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80fe0e0 0x7f1fd810deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fde2c3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd810e4b0 con 0x7f1fd80fe0e0 2024-02-10T06:14:28.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fde2c3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd810ed50 con 0x7f1fd80fcef0 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fd77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80fe0e0 0x7f1fd810deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fd77fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80fe0e0 0x7f1fd810deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39336/0 (socket says 172.21.15.89:39336) 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fd77fe700 1 -- 172.21.15.89:0/2385629122 learned_addr learned my addr 172.21.15.89:0/2385629122 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fd7fff700 1 --2- 172.21.15.89:0/2385629122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fd80fcef0 0x7f1fd810d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fd77fe700 1 -- 172.21.15.89:0/2385629122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fd80fcef0 msgr2=0x7f1fd810d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fd77fe700 1 --2- 172.21.15.89:0/2385629122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fd80fcef0 0x7f1fd810d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fd77fe700 1 -- 172.21.15.89:0/2385629122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc8005040 con 0x7f1fd80fe0e0 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fd77fe700 1 --2- 172.21.15.89:0/2385629122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80fe0e0 0x7f1fd810deb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00c830 tx=0x7f1fcc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.805+0000 7f1fd4ff9700 1 -- 172.21.15.89:0/2385629122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fcc00a760 con 0x7f1fd80fe0e0 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.806+0000 7f1fd4ff9700 1 -- 172.21.15.89:0/2385629122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fcc00ad90 con 0x7f1fd80fe0e0 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.806+0000 7f1fde2c3700 1 -- 172.21.15.89:0/2385629122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd810f030 con 0x7f1fd80fe0e0 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.806+0000 7f1fd4ff9700 1 -- 172.21.15.89:0/2385629122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fcc0113a0 con 0x7f1fd80fe0e0 2024-02-10T06:14:28.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.806+0000 7f1fde2c3700 1 -- 172.21.15.89:0/2385629122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd810f570 con 0x7f1fd80fe0e0 2024-02-10T06:14:28.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.807+0000 7f1fde2c3700 1 -- 172.21.15.89:0/2385629122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd804bed0 con 0x7f1fd80fe0e0 2024-02-10T06:14:28.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.807+0000 7f1fd4ff9700 1 -- 172.21.15.89:0/2385629122 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1fcc012400 con 0x7f1fd80fe0e0 2024-02-10T06:14:28.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.808+0000 7f1fd4ff9700 1 --2- 172.21.15.89:0/2385629122 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1fc00692a0 0x7f1fc006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:28.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.808+0000 7f1fd7fff700 1 --2- 172.21.15.89:0/2385629122 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1fc00692a0 0x7f1fc006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:28.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.808+0000 7f1fd4ff9700 1 -- 172.21.15.89:0/2385629122 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f1fcc087230 con 0x7f1fd80fe0e0 2024-02-10T06:14:28.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.808+0000 7f1fd7fff700 1 --2- 172.21.15.89:0/2385629122 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1fc00692a0 0x7f1fc006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800b3b0 tx=0x7f1fc8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:28.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:28.813+0000 7f1fd4ff9700 1 -- 172.21.15.89:0/2385629122 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1fcc056ff0 con 0x7f1fd80fe0e0 2024-02-10T06:14:29.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:29.049+0000 7f1fde2c3700 1 -- 172.21.15.89:0/2385629122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f1fd8105a30 con 0x7f1fd80fe0e0 2024-02-10T06:14:29.206 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:29 smithi089 ceph-mon[29217]: pgmap v177: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-02-10T06:14:29.399 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:29 smithi173 ceph-mon[36251]: pgmap v177: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-02-10T06:14:30.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.044+0000 7f1fd4ff9700 1 -- 172.21.15.89:0/2385629122 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v8) v1 ==== 121+0+0 (secure 0 0 0) 0x7f1fcc056b80 con 0x7f1fd80fe0e0 2024-02-10T06:14:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.046+0000 7f1fde2c3700 1 -- 172.21.15.89:0/2385629122 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1fc00692a0 msgr2=0x7f1fc006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.046+0000 7f1fde2c3700 1 --2- 172.21.15.89:0/2385629122 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1fc00692a0 0x7f1fc006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800b3b0 tx=0x7f1fc8009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.046+0000 7f1fde2c3700 1 -- 172.21.15.89:0/2385629122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80fe0e0 msgr2=0x7f1fd810deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.046+0000 7f1fde2c3700 1 --2- 172.21.15.89:0/2385629122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80fe0e0 0x7f1fd810deb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00c830 tx=0x7f1fcc004470 comp rx=0 tx=0).stop 2024-02-10T06:14:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.046+0000 7f1fde2c3700 1 -- 172.21.15.89:0/2385629122 shutdown_connections 2024-02-10T06:14:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.046+0000 7f1fde2c3700 1 --2- 172.21.15.89:0/2385629122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fd80fcef0 0x7f1fd810d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.046+0000 7f1fde2c3700 1 --2- 172.21.15.89:0/2385629122 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f1fc00692a0 0x7f1fc006b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.046+0000 7f1fde2c3700 1 --2- 172.21.15.89:0/2385629122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80fe0e0 0x7f1fd810deb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.046+0000 7f1fde2c3700 1 -- 172.21.15.89:0/2385629122 >> 172.21.15.89:0/2385629122 conn(0x7f1fd80f8620 msgr2=0x7f1fd81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.046+0000 7f1fde2c3700 1 -- 172.21.15.89:0/2385629122 shutdown_connections 2024-02-10T06:14:30.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:30.046+0000 7f1fde2c3700 1 -- 172.21.15.89:0/2385629122 wait complete. 2024-02-10T06:14:30.065 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:30 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2385629122' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-10T06:14:30.065 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:30 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:30.065 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:30 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:30.065 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:30 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:30.066 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:30 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi173.cjkmdf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:14:30.066 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:30 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi173.cjkmdf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:14:30.066 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:30 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:30.066 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:30 smithi173 ceph-mon[36251]: Deploying daemon mds.cephfs.smithi173.cjkmdf on smithi173 2024-02-10T06:14:30.326 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:30 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2385629122' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-10T06:14:30.326 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:30 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:30.327 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:30 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:30.327 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:30 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:30.327 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:30 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi173.cjkmdf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T06:14:30.327 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:30 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi173.cjkmdf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T06:14:30.327 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:30 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:30.327 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:30 smithi089 ceph-mon[29217]: Deploying daemon mds.cephfs.smithi173.cjkmdf on smithi173 2024-02-10T06:14:30.636 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:14:30.651 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T06:14:30.651 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-02-10T06:14:30.911 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:31.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:31 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2385629122' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-10T06:14:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:31 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.89:6828/3386828748,v1:172.21.15.89:6829/3386828748] up:boot 2024-02-10T06:14:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:31 smithi089 ceph-mon[29217]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 2 up:standby 2024-02-10T06:14:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:31 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi089.evydlq"}]: dispatch 2024-02-10T06:14:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:31 smithi089 ceph-mon[29217]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 1 up:standby-replay 1 up:standby 2024-02-10T06:14:31.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:31 smithi089 ceph-mon[29217]: pgmap v178: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-10T06:14:31.398 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:31 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2385629122' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-10T06:14:31.398 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:31 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.89:6828/3386828748,v1:172.21.15.89:6829/3386828748] up:boot 2024-02-10T06:14:31.398 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:31 smithi173 ceph-mon[36251]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 2 up:standby 2024-02-10T06:14:31.398 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:31 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi089.evydlq"}]: dispatch 2024-02-10T06:14:31.398 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:31 smithi173 ceph-mon[36251]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 1 up:standby-replay 1 up:standby 2024-02-10T06:14:31.398 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:31 smithi173 ceph-mon[36251]: pgmap v178: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.580+0000 7fe195496700 1 -- 172.21.15.89:0/922298129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1900fe0e0 msgr2=0x7fe1900fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.580+0000 7fe195496700 1 --2- 172.21.15.89:0/922298129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1900fe0e0 0x7fe1900fe550 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe180004940 tx=0x7fe180009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.580+0000 7fe195496700 1 -- 172.21.15.89:0/922298129 shutdown_connections 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.580+0000 7fe195496700 1 --2- 172.21.15.89:0/922298129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1900fe0e0 0x7fe1900fe550 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.580+0000 7fe195496700 1 --2- 172.21.15.89:0/922298129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1900fcef0 0x7fe1900fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.580+0000 7fe195496700 1 -- 172.21.15.89:0/922298129 >> 172.21.15.89:0/922298129 conn(0x7fe1900f8620 msgr2=0x7fe1900faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.580+0000 7fe195496700 1 -- 172.21.15.89:0/922298129 shutdown_connections 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.580+0000 7fe195496700 1 -- 172.21.15.89:0/922298129 wait complete. 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe195496700 1 Processor -- start 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe195496700 1 -- start start 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe195496700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1900fcef0 0x7fe190071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:32.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe195496700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1900fe0e0 0x7fe1900702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe195496700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1900708e0 con 0x7fe1900fe0e0 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe195496700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe190070a50 con 0x7fe1900fcef0 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe18e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1900fe0e0 0x7fe1900702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe18e7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1900fe0e0 0x7fe1900702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39364/0 (socket says 172.21.15.89:39364) 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe18e7fc700 1 -- 172.21.15.89:0/1837213047 learned_addr learned my addr 172.21.15.89:0/1837213047 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe18effd700 1 --2- 172.21.15.89:0/1837213047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1900fcef0 0x7fe190071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe18e7fc700 1 -- 172.21.15.89:0/1837213047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1900fcef0 msgr2=0x7fe190071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe18e7fc700 1 --2- 172.21.15.89:0/1837213047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1900fcef0 0x7fe190071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe18e7fc700 1 -- 172.21.15.89:0/1837213047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe180005040 con 0x7fe1900fe0e0 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.581+0000 7fe18e7fc700 1 --2- 172.21.15.89:0/1837213047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1900fe0e0 0x7fe1900702e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe1800093b0 tx=0x7fe180009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.582+0000 7fe1777fe700 1 -- 172.21.15.89:0/1837213047 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe18000e040 con 0x7fe1900fe0e0 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.582+0000 7fe1777fe700 1 -- 172.21.15.89:0/1837213047 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe18000bcc0 con 0x7fe1900fe0e0 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.582+0000 7fe195496700 1 -- 172.21.15.89:0/1837213047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe190070cd0 con 0x7fe1900fe0e0 2024-02-10T06:14:32.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.582+0000 7fe1777fe700 1 -- 172.21.15.89:0/1837213047 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe18001c630 con 0x7fe1900fe0e0 2024-02-10T06:14:32.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.582+0000 7fe195496700 1 -- 172.21.15.89:0/1837213047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1901159a0 con 0x7fe1900fe0e0 2024-02-10T06:14:32.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.583+0000 7fe195496700 1 -- 172.21.15.89:0/1837213047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19004bed0 con 0x7fe1900fe0e0 2024-02-10T06:14:32.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.583+0000 7fe1777fe700 1 -- 172.21.15.89:0/1837213047 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe180013980 con 0x7fe1900fe0e0 2024-02-10T06:14:32.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.583+0000 7fe1777fe700 1 --2- 172.21.15.89:0/1837213047 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe1780692a0 0x7fe17806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:32.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.584+0000 7fe1777fe700 1 -- 172.21.15.89:0/1837213047 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe180088450 con 0x7fe1900fe0e0 2024-02-10T06:14:32.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.584+0000 7fe18effd700 1 --2- 172.21.15.89:0/1837213047 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe1780692a0 0x7fe17806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:32.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.584+0000 7fe18effd700 1 --2- 172.21.15.89:0/1837213047 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe1780692a0 0x7fe17806b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe190071bf0 tx=0x7fe18400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:32.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.588+0000 7fe1777fe700 1 -- 172.21.15.89:0/1837213047 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe1800582b0 con 0x7fe1900fe0e0 2024-02-10T06:14:32.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:32.824+0000 7fe195496700 1 -- 172.21.15.89:0/1837213047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe190115b60 con 0x7fe1900fe0e0 2024-02-10T06:14:33.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.007+0000 7fe1777fe700 1 -- 172.21.15.89:0/1837213047 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v11) v1 ==== 168+0+0 (secure 0 0 0) 0x7fe180057e40 con 0x7fe1900fe0e0 2024-02-10T06:14:33.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.011+0000 7fe195496700 1 -- 172.21.15.89:0/1837213047 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe1780692a0 msgr2=0x7fe17806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:33.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.011+0000 7fe195496700 1 --2- 172.21.15.89:0/1837213047 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe1780692a0 0x7fe17806b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe190071bf0 tx=0x7fe18400b040 comp rx=0 tx=0).stop 2024-02-10T06:14:33.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.011+0000 7fe195496700 1 -- 172.21.15.89:0/1837213047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1900fe0e0 msgr2=0x7fe1900702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:33.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.011+0000 7fe195496700 1 --2- 172.21.15.89:0/1837213047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1900fe0e0 0x7fe1900702e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe1800093b0 tx=0x7fe180009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:33.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.011+0000 7fe195496700 1 -- 172.21.15.89:0/1837213047 shutdown_connections 2024-02-10T06:14:33.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.011+0000 7fe195496700 1 --2- 172.21.15.89:0/1837213047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1900fcef0 0x7fe190071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:33.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.011+0000 7fe195496700 1 --2- 172.21.15.89:0/1837213047 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe1780692a0 0x7fe17806b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:33.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.011+0000 7fe195496700 1 --2- 172.21.15.89:0/1837213047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1900fe0e0 0x7fe1900702e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:33.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.011+0000 7fe195496700 1 -- 172.21.15.89:0/1837213047 >> 172.21.15.89:0/1837213047 conn(0x7fe1900f8620 msgr2=0x7fe1901013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:33.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.012+0000 7fe195496700 1 -- 172.21.15.89:0/1837213047 shutdown_connections 2024-02-10T06:14:33.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:33.012+0000 7fe195496700 1 -- 172.21.15.89:0/1837213047 wait complete. 2024-02-10T06:14:33.013 INFO:teuthology.orchestra.run.smithi089.stderr:inline data enabled 2024-02-10T06:14:33.028 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:32 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.028 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:32 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.028 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:32 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.028 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:32 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.028 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:32 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.028 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:32 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:33.028 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:32 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.173:6826/2798078299,v1:172.21.15.173:6827/2798078299] up:boot 2024-02-10T06:14:33.028 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:32 smithi089 ceph-mon[29217]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 1 up:standby-replay 2 up:standby 2024-02-10T06:14:33.028 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:32 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi173.cjkmdf"}]: dispatch 2024-02-10T06:14:33.028 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:32 smithi089 ceph-mon[29217]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:14:33.029 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:32 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.144 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:32 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.144 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:32 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.144 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:32 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.144 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:32 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.145 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:32 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.145 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:32 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:33.145 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:32 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.173:6826/2798078299,v1:172.21.15.173:6827/2798078299] up:boot 2024-02-10T06:14:33.145 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:32 smithi173 ceph-mon[36251]: fsmap cephfs:1 {0=cephfs.smithi089.mgkvmj=up:active} 1 up:standby-replay 2 up:standby 2024-02-10T06:14:33.145 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:32 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi173.cjkmdf"}]: dispatch 2024-02-10T06:14:33.145 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:32 smithi173 ceph-mon[36251]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T06:14:33.145 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:32 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:33 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1837213047' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:14:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:33 smithi089 ceph-mon[29217]: Dropping low affinity active daemon mds.cephfs.smithi089.mgkvmj in favor of higher affinity standby. 2024-02-10T06:14:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:33 smithi089 ceph-mon[29217]: Replacing daemon mds.cephfs.smithi089.mgkvmj as rank 0 with standby daemon mds.cephfs.smithi173.tumnkz 2024-02-10T06:14:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:33 smithi089 ceph-mon[29217]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T06:14:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:33 smithi089 ceph-mon[29217]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-10T06:14:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:33 smithi089 ceph-mon[29217]: osdmap e47: 6 total, 6 up, 6 in 2024-02-10T06:14:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:33 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] up:standby-replay 2024-02-10T06:14:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:33 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1837213047' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:14:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:33 smithi089 ceph-mon[29217]: fsmap cephfs:1/1 {0=cephfs.smithi173.tumnkz=up:replay} 2 up:standby 2024-02-10T06:14:33.916 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T06:14:33.927 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T06:14:33.927 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-10T06:14:34.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:33 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/1837213047' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:14:34.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:33 smithi173 ceph-mon[36251]: Dropping low affinity active daemon mds.cephfs.smithi089.mgkvmj in favor of higher affinity standby. 2024-02-10T06:14:34.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:33 smithi173 ceph-mon[36251]: Replacing daemon mds.cephfs.smithi089.mgkvmj as rank 0 with standby daemon mds.cephfs.smithi173.tumnkz 2024-02-10T06:14:34.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:33 smithi173 ceph-mon[36251]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T06:14:34.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:33 smithi173 ceph-mon[36251]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-10T06:14:34.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:33 smithi173 ceph-mon[36251]: osdmap e47: 6 total, 6 up, 6 in 2024-02-10T06:14:34.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:33 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] up:standby-replay 2024-02-10T06:14:34.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:33 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/1837213047' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:14:34.150 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:33 smithi173 ceph-mon[36251]: fsmap cephfs:1/1 {0=cephfs.smithi173.tumnkz=up:replay} 2 up:standby 2024-02-10T06:14:34.639 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:35.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:34 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:35.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:34 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:35.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:34 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.89:6826/3574254294,v1:172.21.15.89:6827/3574254294] up:boot 2024-02-10T06:14:35.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:34 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] up:reconnect 2024-02-10T06:14:35.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:34 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.89:6828/3386828748,v1:172.21.15.89:6829/3386828748] up:standby 2024-02-10T06:14:35.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:34 smithi089 ceph-mon[29217]: fsmap cephfs:1/1 {0=cephfs.smithi173.tumnkz=up:reconnect} 3 up:standby 2024-02-10T06:14:35.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:34 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi089.mgkvmj"}]: dispatch 2024-02-10T06:14:35.105 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:34 smithi089 ceph-mon[29217]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-10T06:14:35.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:34 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:35.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:34 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:35.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:34 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.89:6826/3574254294,v1:172.21.15.89:6827/3574254294] up:boot 2024-02-10T06:14:35.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:34 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] up:reconnect 2024-02-10T06:14:35.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:34 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.89:6828/3386828748,v1:172.21.15.89:6829/3386828748] up:standby 2024-02-10T06:14:35.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:34 smithi173 ceph-mon[36251]: fsmap cephfs:1/1 {0=cephfs.smithi173.tumnkz=up:reconnect} 3 up:standby 2024-02-10T06:14:35.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:34 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi089.mgkvmj"}]: dispatch 2024-02-10T06:14:35.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:34 smithi173 ceph-mon[36251]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-10T06:14:36.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:36 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] up:rejoin 2024-02-10T06:14:36.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:36 smithi089 ceph-mon[29217]: fsmap cephfs:1/1 {0=cephfs.smithi173.tumnkz=up:rejoin} 3 up:standby 2024-02-10T06:14:36.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:36 smithi089 ceph-mon[29217]: daemon mds.cephfs.smithi173.tumnkz is now active in filesystem cephfs as rank 0 2024-02-10T06:14:36.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:36 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] up:rejoin 2024-02-10T06:14:36.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:36 smithi173 ceph-mon[36251]: fsmap cephfs:1/1 {0=cephfs.smithi173.tumnkz=up:rejoin} 3 up:standby 2024-02-10T06:14:36.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:36 smithi173 ceph-mon[36251]: daemon mds.cephfs.smithi173.tumnkz is now active in filesystem cephfs as rank 0 2024-02-10T06:14:37.323 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:37 smithi089 ceph-mon[29217]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T06:14:37.323 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:37 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] up:active 2024-02-10T06:14:37.323 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:37 smithi089 ceph-mon[29217]: mds.? [v2:172.21.15.173:6826/2798078299,v1:172.21.15.173:6827/2798078299] up:standby 2024-02-10T06:14:37.323 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:37 smithi089 ceph-mon[29217]: fsmap cephfs:1 {0=cephfs.smithi173.tumnkz=up:active} 3 up:standby 2024-02-10T06:14:37.323 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:37 smithi089 ceph-mon[29217]: fsmap cephfs:1 {0=cephfs.smithi173.tumnkz=up:active} 1 up:standby-replay 2 up:standby 2024-02-10T06:14:37.324 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:37 smithi089 ceph-mon[29217]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 716 B/s wr, 4 op/s 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.321+0000 7f804ac41700 1 -- 172.21.15.89:0/423926813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80440fbe40 msgr2=0x7f80440fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.321+0000 7f804ac41700 1 --2- 172.21.15.89:0/423926813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80440fbe40 0x7f80440fc250 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8038004940 tx=0x7f8038009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.321+0000 7f804ac41700 1 -- 172.21.15.89:0/423926813 shutdown_connections 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.321+0000 7f804ac41700 1 --2- 172.21.15.89:0/423926813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80440fcfb0 0x7f80440fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.321+0000 7f804ac41700 1 --2- 172.21.15.89:0/423926813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80440fbe40 0x7f80440fc250 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.321+0000 7f804ac41700 1 -- 172.21.15.89:0/423926813 >> 172.21.15.89:0/423926813 conn(0x7f80440f74e0 msgr2=0x7f80440f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.321+0000 7f804ac41700 1 -- 172.21.15.89:0/423926813 shutdown_connections 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.321+0000 7f804ac41700 1 -- 172.21.15.89:0/423926813 wait complete. 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.322+0000 7f804ac41700 1 Processor -- start 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.322+0000 7f804ac41700 1 -- start start 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.322+0000 7f804ac41700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80440fbe40 0x7f8044110600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f804ac41700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80440fcfb0 0x7f8044112b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f80489dd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80440fbe40 0x7f8044110600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f80489dd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80440fbe40 0x7f8044110600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39392/0 (socket says 172.21.15.89:39392) 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f80489dd700 1 -- 172.21.15.89:0/1219610477 learned_addr learned my addr 172.21.15.89:0/1219610477 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80440664c0 con 0x7f80440fbe40 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8044066600 con 0x7f80440fcfb0 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f8043fff700 1 --2- 172.21.15.89:0/1219610477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80440fcfb0 0x7f8044112b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f80489dd700 1 -- 172.21.15.89:0/1219610477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80440fcfb0 msgr2=0x7f8044112b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f80489dd700 1 --2- 172.21.15.89:0/1219610477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80440fcfb0 0x7f8044112b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f80489dd700 1 -- 172.21.15.89:0/1219610477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8038005040 con 0x7f80440fbe40 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f80489dd700 1 --2- 172.21.15.89:0/1219610477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80440fbe40 0x7f8044110600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f80380093e0 tx=0x7f8038009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f80417fa700 1 -- 172.21.15.89:0/1219610477 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f803800e040 con 0x7f80440fbe40 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f80417fa700 1 -- 172.21.15.89:0/1219610477 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f803800bcc0 con 0x7f80440fbe40 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8044113150 con 0x7f80440fbe40 2024-02-10T06:14:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f80417fa700 1 -- 172.21.15.89:0/1219610477 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f803801c630 con 0x7f80440fbe40 2024-02-10T06:14:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.323+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8044113690 con 0x7f80440fbe40 2024-02-10T06:14:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.325+0000 7f80417fa700 1 -- 172.21.15.89:0/1219610477 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f803801c8a0 con 0x7f80440fbe40 2024-02-10T06:14:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.325+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f804404bed0 con 0x7f80440fbe40 2024-02-10T06:14:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.325+0000 7f80417fa700 1 --2- 172.21.15.89:0/1219610477 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f80340692a0 0x7f803406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.326+0000 7f8043fff700 1 --2- 172.21.15.89:0/1219610477 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f80340692a0 0x7f803406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.326+0000 7f80417fa700 1 -- 172.21.15.89:0/1219610477 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8038013780 con 0x7f80440fbe40 2024-02-10T06:14:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.326+0000 7f8043fff700 1 --2- 172.21.15.89:0/1219610477 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f80340692a0 0x7f803406b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8030007fd0 tx=0x7f8030009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.330+0000 7f80417fa700 1 -- 172.21.15.89:0/1219610477 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8038055930 con 0x7f80440fbe40 2024-02-10T06:14:37.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:37 smithi173 ceph-mon[36251]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T06:14:37.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:37 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] up:active 2024-02-10T06:14:37.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:37 smithi173 ceph-mon[36251]: mds.? [v2:172.21.15.173:6826/2798078299,v1:172.21.15.173:6827/2798078299] up:standby 2024-02-10T06:14:37.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:37 smithi173 ceph-mon[36251]: fsmap cephfs:1 {0=cephfs.smithi173.tumnkz=up:active} 3 up:standby 2024-02-10T06:14:37.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:37 smithi173 ceph-mon[36251]: fsmap cephfs:1 {0=cephfs.smithi173.tumnkz=up:active} 1 up:standby-replay 2 up:standby 2024-02-10T06:14:37.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:37 smithi173 ceph-mon[36251]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 716 B/s wr, 4 op/s 2024-02-10T06:14:37.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.570+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8044113aa0 con 0x7f80440fbe40 2024-02-10T06:14:37.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.571+0000 7f80417fa700 1 -- 172.21.15.89:0/1219610477 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1838 (secure 0 0 0) 0x7f8038058f50 con 0x7f80440fbe40 2024-02-10T06:14:37.572 INFO:teuthology.orchestra.run.smithi089.stdout:e15 2024-02-10T06:14:37.572 INFO:teuthology.orchestra.run.smithi089.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:14:37.572 INFO:teuthology.orchestra.run.smithi089.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:14:37.572 INFO:teuthology.orchestra.run.smithi089.stdout:legacy client fscid: 1 2024-02-10T06:14:37.572 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:14:37.572 INFO:teuthology.orchestra.run.smithi089.stdout:Filesystem 'cephfs' (1) 2024-02-10T06:14:37.572 INFO:teuthology.orchestra.run.smithi089.stdout:fs_name cephfs 2024-02-10T06:14:37.572 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 15 2024-02-10T06:14:37.572 INFO:teuthology.orchestra.run.smithi089.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-10T06:14:37.572 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-02-10T06:14:18.236041+0000 2024-02-10T06:14:37.572 INFO:teuthology.orchestra.run.smithi089.stdout:modified 2024-02-10T06:14:36.018945+0000 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:tableserver 0 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:root 0 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:session_timeout 60 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:session_autoclose 300 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:max_file_size 1099511627776 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:required_client_features {} 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure 0 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure_osd_epoch 47 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:max_mds 1 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:in 0 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:up {0=24235} 2024-02-10T06:14:37.573 INFO:teuthology.orchestra.run.smithi089.stdout:failed 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:damaged 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:stopped 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:data_pools [3] 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:metadata_pool 2 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:inline_data enabled 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:balancer 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:standby_count_wanted 1 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi173.tumnkz{0:24235} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi089.evydlq{0:14476} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.89:6828/3386828748,v1:172.21.15.89:6829/3386828748] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:Standby daemons: 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:14:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi173.cjkmdf{-1:24243} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.173:6826/2798078299,v1:172.21.15.173:6827/2798078299] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:14:37.575 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi089.mgkvmj{-1:24251} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.89:6826/3574254294,v1:172.21.15.89:6827/3574254294] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:14:37.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.573+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f80340692a0 msgr2=0x7f803406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:37.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.573+0000 7f804ac41700 1 --2- 172.21.15.89:0/1219610477 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f80340692a0 0x7f803406b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8030007fd0 tx=0x7f8030009040 comp rx=0 tx=0).stop 2024-02-10T06:14:37.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.573+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80440fbe40 msgr2=0x7f8044110600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:37.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.573+0000 7f804ac41700 1 --2- 172.21.15.89:0/1219610477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80440fbe40 0x7f8044110600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f80380093e0 tx=0x7f8038009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:37.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.573+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 shutdown_connections 2024-02-10T06:14:37.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.573+0000 7f804ac41700 1 --2- 172.21.15.89:0/1219610477 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f80340692a0 0x7f803406b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:37.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.573+0000 7f804ac41700 1 --2- 172.21.15.89:0/1219610477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80440fbe40 0x7f8044110600 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:37.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.573+0000 7f804ac41700 1 --2- 172.21.15.89:0/1219610477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80440fcfb0 0x7f8044112b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:37.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.573+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 >> 172.21.15.89:0/1219610477 conn(0x7f80440f74e0 msgr2=0x7f8044100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:37.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.573+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 shutdown_connections 2024-02-10T06:14:37.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:37.573+0000 7f804ac41700 1 -- 172.21.15.89:0/1219610477 wait complete. 2024-02-10T06:14:37.576 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 15 2024-02-10T06:14:38.312 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:38 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1219610477' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:14:38.337 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-02-10T06:14:38.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:38 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/1219610477' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:14:38.682 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:39.057 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:39 smithi089 ceph-mon[29217]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-10T06:14:39.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:39 smithi173 ceph-mon[36251]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-10T06:14:40.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.290+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/640238205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d340fbce0 msgr2=0x7f7d34102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:40.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.290+0000 7f7d3ba2b700 1 --2- 172.21.15.89:0/640238205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d340fbce0 0x7f7d34102560 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7d24004940 tx=0x7f7d24009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:40.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.290+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/640238205 shutdown_connections 2024-02-10T06:14:40.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.290+0000 7f7d3ba2b700 1 --2- 172.21.15.89:0/640238205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d340fbce0 0x7f7d34102560 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:40.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.290+0000 7f7d3ba2b700 1 --2- 172.21.15.89:0/640238205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340fb300 0x7f7d340fb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:40.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.290+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/640238205 >> 172.21.15.89:0/640238205 conn(0x7f7d340f6ca0 msgr2=0x7f7d340f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:40.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.291+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/640238205 shutdown_connections 2024-02-10T06:14:40.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.291+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/640238205 wait complete. 2024-02-10T06:14:40.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.291+0000 7f7d3ba2b700 1 Processor -- start 2024-02-10T06:14:40.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d3ba2b700 1 -- start start 2024-02-10T06:14:40.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d3ba2b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340fb300 0x7f7d3410da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:40.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d3ba2b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d3410df50 0x7f7d3410edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d3ba2b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3410e410 con 0x7f7d3410df50 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d3ba2b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3410e550 con 0x7f7d340fb300 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d38fc6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d3410df50 0x7f7d3410edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d38fc6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d3410df50 0x7f7d3410edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34694/0 (socket says 172.21.15.89:34694) 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d38fc6700 1 -- 172.21.15.89:0/891182988 learned_addr learned my addr 172.21.15.89:0/891182988 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d397c7700 1 --2- 172.21.15.89:0/891182988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340fb300 0x7f7d3410da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d38fc6700 1 -- 172.21.15.89:0/891182988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340fb300 msgr2=0x7f7d3410da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d38fc6700 1 --2- 172.21.15.89:0/891182988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340fb300 0x7f7d3410da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.292+0000 7f7d38fc6700 1 -- 172.21.15.89:0/891182988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d24005040 con 0x7f7d3410df50 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.293+0000 7f7d38fc6700 1 --2- 172.21.15.89:0/891182988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d3410df50 0x7f7d3410edf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009150 tx=0x7f7d24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.293+0000 7f7d29ffb700 1 -- 172.21.15.89:0/891182988 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d2400e040 con 0x7f7d3410df50 2024-02-10T06:14:40.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.293+0000 7f7d29ffb700 1 -- 172.21.15.89:0/891182988 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d2400bbf0 con 0x7f7d3410df50 2024-02-10T06:14:40.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.293+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/891182988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3410f3f0 con 0x7f7d3410df50 2024-02-10T06:14:40.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.293+0000 7f7d29ffb700 1 -- 172.21.15.89:0/891182988 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d2401c3f0 con 0x7f7d3410df50 2024-02-10T06:14:40.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.293+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/891182988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3410f900 con 0x7f7d3410df50 2024-02-10T06:14:40.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.294+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/891182988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d34107a90 con 0x7f7d3410df50 2024-02-10T06:14:40.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.294+0000 7f7d29ffb700 1 -- 172.21.15.89:0/891182988 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7d24013420 con 0x7f7d3410df50 2024-02-10T06:14:40.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.295+0000 7f7d29ffb700 1 --2- 172.21.15.89:0/891182988 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7d200692a0 0x7f7d2006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:40.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.295+0000 7f7d29ffb700 1 -- 172.21.15.89:0/891182988 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7d240885f0 con 0x7f7d3410df50 2024-02-10T06:14:40.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.295+0000 7f7d397c7700 1 --2- 172.21.15.89:0/891182988 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7d200692a0 0x7f7d2006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:40.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.295+0000 7f7d397c7700 1 --2- 172.21.15.89:0/891182988 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7d200692a0 0x7f7d2006b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7d34065f20 tx=0x7f7d3000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:40.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.299+0000 7f7d29ffb700 1 -- 172.21.15.89:0/891182988 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7d24058360 con 0x7f7d3410df50 2024-02-10T06:14:40.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.539+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/891182988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d3410fc50 con 0x7f7d3410df50 2024-02-10T06:14:40.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.540+0000 7f7d29ffb700 1 -- 172.21.15.89:0/891182988 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4702 (secure 0 0 0) 0x7f7d24057ef0 con 0x7f7d3410df50 2024-02-10T06:14:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.542+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/891182988 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7d200692a0 msgr2=0x7f7d2006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.542+0000 7f7d3ba2b700 1 --2- 172.21.15.89:0/891182988 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7d200692a0 0x7f7d2006b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7d34065f20 tx=0x7f7d3000b040 comp rx=0 tx=0).stop 2024-02-10T06:14:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.542+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/891182988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d3410df50 msgr2=0x7f7d3410edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:40.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.542+0000 7f7d3ba2b700 1 --2- 172.21.15.89:0/891182988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d3410df50 0x7f7d3410edf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009150 tx=0x7f7d24009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.542+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/891182988 shutdown_connections 2024-02-10T06:14:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.542+0000 7f7d3ba2b700 1 --2- 172.21.15.89:0/891182988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d340fb300 0x7f7d3410da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.542+0000 7f7d3ba2b700 1 --2- 172.21.15.89:0/891182988 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7d200692a0 0x7f7d2006b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.542+0000 7f7d3ba2b700 1 --2- 172.21.15.89:0/891182988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7d3410df50 0x7f7d3410edf0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.542+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/891182988 >> 172.21.15.89:0/891182988 conn(0x7f7d340f6ca0 msgr2=0x7f7d34100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.542+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/891182988 shutdown_connections 2024-02-10T06:14:40.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:40.542+0000 7f7d3ba2b700 1 -- 172.21.15.89:0/891182988 wait complete. 2024-02-10T06:14:40.546 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 15 2024-02-10T06:14:40.554 INFO:teuthology.orchestra.run.smithi089.stdout:true 2024-02-10T06:14:40.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:40 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:40.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:40 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:40.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:40 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:40.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:40 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:40.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:40 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:40.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:40 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:40.850 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:40 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:40.850 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:40 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:40.850 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:40 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:40.850 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:40 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:40.850 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:40 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:40.850 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:40 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:41.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:41 smithi089 ceph-mon[29217]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:14:41.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:41 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/891182988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-10T06:14:41.695 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-02-10T06:14:41.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:41 smithi173 ceph-mon[36251]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:14:41.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:41 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/891182988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-10T06:14:42.399 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:42.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:42.803 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:42.804 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:42.804 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:42.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:42.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:42.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:42.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:43.712 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:43 smithi089 ceph-mon[29217]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:14:43.900 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:43 smithi173 ceph-mon[36251]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:14:45.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.747+0000 7fe2a1634700 1 -- 172.21.15.89:0/1428866555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe29c0fe0e0 msgr2=0x7fe29c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:45.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.747+0000 7fe2a1634700 1 --2- 172.21.15.89:0/1428866555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe29c0fe0e0 0x7fe29c0fe550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe28c004940 tx=0x7fe28c009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:45.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.747+0000 7fe2a1634700 1 -- 172.21.15.89:0/1428866555 shutdown_connections 2024-02-10T06:14:45.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.747+0000 7fe2a1634700 1 --2- 172.21.15.89:0/1428866555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe29c0fe0e0 0x7fe29c0fe550 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:45.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.747+0000 7fe2a1634700 1 --2- 172.21.15.89:0/1428866555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe29c0fcef0 0x7fe29c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:45.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.747+0000 7fe2a1634700 1 -- 172.21.15.89:0/1428866555 >> 172.21.15.89:0/1428866555 conn(0x7fe29c0f8620 msgr2=0x7fe29c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:45.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.747+0000 7fe2a1634700 1 -- 172.21.15.89:0/1428866555 shutdown_connections 2024-02-10T06:14:45.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.747+0000 7fe2a1634700 1 -- 172.21.15.89:0/1428866555 wait complete. 2024-02-10T06:14:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.748+0000 7fe2a1634700 1 Processor -- start 2024-02-10T06:14:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.748+0000 7fe2a1634700 1 -- start start 2024-02-10T06:14:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.748+0000 7fe2a1634700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe29c0fcef0 0x7fe29c071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.748+0000 7fe2a1634700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe29c0fe0e0 0x7fe29c0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:45.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.748+0000 7fe2a1634700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c070970 con 0x7fe29c0fe0e0 2024-02-10T06:14:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.748+0000 7fe2a1634700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c070ae0 con 0x7fe29c0fcef0 2024-02-10T06:14:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.748+0000 7fe29a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe29c0fe0e0 0x7fe29c0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.748+0000 7fe29a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe29c0fe0e0 0x7fe29c0702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34710/0 (socket says 172.21.15.89:34710) 2024-02-10T06:14:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.748+0000 7fe29a7fc700 1 -- 172.21.15.89:0/184700042 learned_addr learned my addr 172.21.15.89:0/184700042 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.748+0000 7fe29affd700 1 --2- 172.21.15.89:0/184700042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe29c0fcef0 0x7fe29c071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.749+0000 7fe29a7fc700 1 -- 172.21.15.89:0/184700042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe29c0fcef0 msgr2=0x7fe29c071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.749+0000 7fe29a7fc700 1 --2- 172.21.15.89:0/184700042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe29c0fcef0 0x7fe29c071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:45.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.749+0000 7fe29a7fc700 1 -- 172.21.15.89:0/184700042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c005040 con 0x7fe29c0fe0e0 2024-02-10T06:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.749+0000 7fe29a7fc700 1 --2- 172.21.15.89:0/184700042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe29c0fe0e0 0x7fe29c0702e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0093b0 tx=0x7fe28c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.749+0000 7fe2837fe700 1 -- 172.21.15.89:0/184700042 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe28c00e040 con 0x7fe29c0fe0e0 2024-02-10T06:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.749+0000 7fe2837fe700 1 -- 172.21.15.89:0/184700042 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe28c00bcc0 con 0x7fe29c0fe0e0 2024-02-10T06:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.749+0000 7fe2837fe700 1 -- 172.21.15.89:0/184700042 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe28c01c630 con 0x7fe29c0fe0e0 2024-02-10T06:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.749+0000 7fe2a1634700 1 -- 172.21.15.89:0/184700042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29c070d60 con 0x7fe29c0fe0e0 2024-02-10T06:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.749+0000 7fe2a1634700 1 -- 172.21.15.89:0/184700042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c1159c0 con 0x7fe29c0fe0e0 2024-02-10T06:14:45.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.750+0000 7fe2a1634700 1 -- 172.21.15.89:0/184700042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29c04bed0 con 0x7fe29c0fe0e0 2024-02-10T06:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.750+0000 7fe2837fe700 1 -- 172.21.15.89:0/184700042 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe28c0139f0 con 0x7fe29c0fe0e0 2024-02-10T06:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.751+0000 7fe2837fe700 1 --2- 172.21.15.89:0/184700042 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe2840692a0 0x7fe28406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.751+0000 7fe2837fe700 1 -- 172.21.15.89:0/184700042 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe28c0885e0 con 0x7fe29c0fe0e0 2024-02-10T06:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.751+0000 7fe29affd700 1 --2- 172.21.15.89:0/184700042 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe2840692a0 0x7fe28406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:45.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.751+0000 7fe29affd700 1 --2- 172.21.15.89:0/184700042 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe2840692a0 0x7fe28406b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe290006fd0 tx=0x7fe290008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:45.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:45.755+0000 7fe2837fe700 1 -- 172.21.15.89:0/184700042 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe28c058350 con 0x7fe29c0fe0e0 2024-02-10T06:14:45.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:45 smithi089 ceph-mon[29217]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:14:45.903 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:45 smithi173 ceph-mon[36251]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:14:46.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.039+0000 7fe2a1634700 1 -- 172.21.15.89:0/184700042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fe29c115d90 con 0x7fe29c0fe0e0 2024-02-10T06:14:46.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.040+0000 7fe2837fe700 1 -- 172.21.15.89:0/184700042 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v15) v1 ==== 78+0+99 (secure 0 0 0) 0x7fe28c057ee0 con 0x7fe29c0fe0e0 2024-02-10T06:14:46.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.042+0000 7fe2a1634700 1 -- 172.21.15.89:0/184700042 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe2840692a0 msgr2=0x7fe28406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:46.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.042+0000 7fe2a1634700 1 --2- 172.21.15.89:0/184700042 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe2840692a0 0x7fe28406b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe290006fd0 tx=0x7fe290008040 comp rx=0 tx=0).stop 2024-02-10T06:14:46.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.042+0000 7fe2a1634700 1 -- 172.21.15.89:0/184700042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe29c0fe0e0 msgr2=0x7fe29c0702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:46.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.042+0000 7fe2a1634700 1 --2- 172.21.15.89:0/184700042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe29c0fe0e0 0x7fe29c0702e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0093b0 tx=0x7fe28c009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:46.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.042+0000 7fe2a1634700 1 -- 172.21.15.89:0/184700042 shutdown_connections 2024-02-10T06:14:46.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.042+0000 7fe2a1634700 1 --2- 172.21.15.89:0/184700042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe29c0fcef0 0x7fe29c071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:46.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.042+0000 7fe2a1634700 1 --2- 172.21.15.89:0/184700042 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe2840692a0 0x7fe28406b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:46.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.042+0000 7fe2a1634700 1 --2- 172.21.15.89:0/184700042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe29c0fe0e0 0x7fe29c0702e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:46.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.042+0000 7fe2a1634700 1 -- 172.21.15.89:0/184700042 >> 172.21.15.89:0/184700042 conn(0x7fe29c0f8620 msgr2=0x7fe29c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:46.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.042+0000 7fe2a1634700 1 -- 172.21.15.89:0/184700042 shutdown_connections 2024-02-10T06:14:46.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.042+0000 7fe2a1634700 1 -- 172.21.15.89:0/184700042 wait complete. 2024-02-10T06:14:46.061 INFO:teuthology.orchestra.run.smithi089.stdout:true 2024-02-10T06:14:46.678 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-02-10T06:14:46.711 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-10T06:14:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.828+0000 7f7bab99d640 1 -- 172.21.15.89:0/1063386636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ba40ff430 msgr2=0x7f7ba4101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.828+0000 7f7bab99d640 1 --2- 172.21.15.89:0/1063386636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ba40ff430 0x7f7ba4101810 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7b940048d0 tx=0x7f7b94009da0 comp rx=0 tx=0).stop 2024-02-10T06:14:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.828+0000 7f7bab99d640 1 -- 172.21.15.89:0/1063386636 shutdown_connections 2024-02-10T06:14:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.828+0000 7f7bab99d640 1 --2- 172.21.15.89:0/1063386636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ba40ff430 0x7f7ba4101810 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.828+0000 7f7bab99d640 1 --2- 172.21.15.89:0/1063386636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ba4066350 0x7f7ba40feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.828+0000 7f7bab99d640 1 -- 172.21.15.89:0/1063386636 >> 172.21.15.89:0/1063386636 conn(0x7f7ba40f6940 msgr2=0x7f7ba40f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.828+0000 7f7bab99d640 1 -- 172.21.15.89:0/1063386636 shutdown_connections 2024-02-10T06:14:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.828+0000 7f7bab99d640 1 -- 172.21.15.89:0/1063386636 wait complete. 2024-02-10T06:14:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7bab99d640 1 Processor -- start 2024-02-10T06:14:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7bab99d640 1 -- start start 2024-02-10T06:14:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7bab99d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ba4066350 0x7f7ba410cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7bab99d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ba40ff430 0x7f7ba410d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7bab99d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba410d6f0 con 0x7f7ba40ff430 2024-02-10T06:14:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7bab99d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba4112140 con 0x7f7ba4066350 2024-02-10T06:14:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7ba8f38640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ba40ff430 0x7f7ba410d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7ba8f38640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ba40ff430 0x7f7ba410d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34728/0 (socket says 172.21.15.89:34728) 2024-02-10T06:14:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7ba8f38640 1 -- 172.21.15.89:0/432636092 learned_addr learned my addr 172.21.15.89:0/432636092 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7ba8f38640 1 -- 172.21.15.89:0/432636092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ba4066350 msgr2=0x7f7ba410cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7ba8f38640 1 --2- 172.21.15.89:0/432636092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ba4066350 0x7f7ba410cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7ba8f38640 1 -- 172.21.15.89:0/432636092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b94005040 con 0x7f7ba40ff430 2024-02-10T06:14:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7ba8f38640 1 --2- 172.21.15.89:0/432636092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ba40ff430 0x7f7ba410d0a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7b940043d0 tx=0x7f7b94009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7b99ffb640 1 -- 172.21.15.89:0/432636092 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b940092e0 con 0x7f7ba40ff430 2024-02-10T06:14:46.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7b99ffb640 1 -- 172.21.15.89:0/432636092 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b94009440 con 0x7f7ba40ff430 2024-02-10T06:14:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.829+0000 7f7bab99d640 1 -- 172.21.15.89:0/432636092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba4112280 con 0x7f7ba40ff430 2024-02-10T06:14:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.830+0000 7f7b99ffb640 1 -- 172.21.15.89:0/432636092 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b9403a680 con 0x7f7ba40ff430 2024-02-10T06:14:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.830+0000 7f7bab99d640 1 -- 172.21.15.89:0/432636092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba4112770 con 0x7f7ba40ff430 2024-02-10T06:14:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.831+0000 7f7bab99d640 1 -- 172.21.15.89:0/432636092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba4104de0 con 0x7f7ba40ff430 2024-02-10T06:14:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.831+0000 7f7b99ffb640 1 -- 172.21.15.89:0/432636092 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7b9403b420 con 0x7f7ba40ff430 2024-02-10T06:14:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.831+0000 7f7b99ffb640 1 --2- 172.21.15.89:0/432636092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7b90069210 0x7f7b9006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.831+0000 7f7b99ffb640 1 -- 172.21.15.89:0/432636092 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7b940a9fa0 con 0x7f7ba40ff430 2024-02-10T06:14:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.831+0000 7f7ba9739640 1 --2- 172.21.15.89:0/432636092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7b90069210 0x7f7b9006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.832+0000 7f7ba9739640 1 --2- 172.21.15.89:0/432636092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7b90069210 0x7f7b9006b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0009af0 tx=0x7f7ba0009cd0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:46.836+0000 7f7b99ffb640 1 -- 172.21.15.89:0/432636092 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7b94076280 con 0x7f7ba40ff430 2024-02-10T06:14:47.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.001+0000 7f7bab99d640 1 -- 172.21.15.89:0/432636092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ba4112a80 con 0x7f7ba40ff430 2024-02-10T06:14:47.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.001+0000 7f7b99ffb640 1 -- 172.21.15.89:0/432636092 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14373 (secure 0 0 0) 0x7f7b940798a0 con 0x7f7ba40ff430 2024-02-10T06:14:47.002 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:14:47.002 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":47,"fsid":"84e5b1ba-c7da-11ee-95b8-87774f69a715","created":"2024-02-10T06:07:41.156060+0000","modified":"2024-02-10T06:14:33.003240+0000","last_up_change":"2024-02-10T06:13:03.312644+0000","last_in_change":"2024-02-10T06:12:45.108728+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T06:11:52.416365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-10T06:14:16.816453+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-10T06:14:17.466421+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fae919bb-103f-402d-8509-1e29a0ebc746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6803","nonce":2704721040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6805","nonce":2704721040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6809","nonce":2704721040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2704721040},{"type":"v1","addr":"172.21.15.89:6807","nonce":2704721040}]},"public_addr":"172.21.15.89:6803/2704721040","cluster_addr":"172.21.15.89:6805/2704721040","heartbeat_back_addr":"172.21.15.89:6809/2704721040","heartbeat_front_addr":"172.21.15.89:6807/2704721040","state":["exists","up"]},{"osd":1,"uuid":"61122d5c-670d-4d4f-8575-d3759758e348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6811","nonce":3051839828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6813","nonce":3051839828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6817","nonce":3051839828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":3051839828},{"type":"v1","addr":"172.21.15.89:6815","nonce":3051839828}]},"public_addr":"172.21.15.89:6811/3051839828","cluster_addr":"172.21.15.89:6813/3051839828","heartbeat_back_addr":"172.21.15.89:6817/3051839828","heartbeat_front_addr":"172.21.15.89:6815/3051839828","state":["exists","up"]},{"osd":2,"uuid":"3ffd67bc-829e-449f-b4c3-94a11da55da6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6819","nonce":637372379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6821","nonce":637372379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6825","nonce":637372379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":637372379},{"type":"v1","addr":"172.21.15.89:6823","nonce":637372379}]},"public_addr":"172.21.15.89:6819/637372379","cluster_addr":"172.21.15.89:6821/637372379","heartbeat_back_addr":"172.21.15.89:6825/637372379","heartbeat_front_addr":"172.21.15.89:6823/637372379","state":["exists","up"]},{"osd":3,"uuid":"cc150631-29f2-4b5d-846d-bffbe2c09cec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6801","nonce":2644357236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6803","nonce":2644357236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6807","nonce":2644357236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2644357236},{"type":"v1","addr":"172.21.15.173:6805","nonce":2644357236}]},"public_addr":"172.21.15.173:6801/2644357236","cluster_addr":"172.21.15.173:6803/2644357236","heartbeat_back_addr":"172.21.15.173:6807/2644357236","heartbeat_front_addr":"172.21.15.173:6805/2644357236","state":["exists","up"]},{"osd":4,"uuid":"cccd0c3a-15fc-43e4-852d-f392f991ef61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6809","nonce":2725914780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6811","nonce":2725914780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6815","nonce":2725914780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2725914780},{"type":"v1","addr":"172.21.15.173:6813","nonce":2725914780}]},"public_addr":"172.21.15.173:6809/2725914780","cluster_addr":"172.21.15.173:6811/2725914780","heartbeat_back_addr":"172.21.15.173:6815/2725914780","heartbeat_front_addr":"172.21.15.173:6813/2725914780","state":["exists","up"]},{"osd":5,"uuid":"00386df2-a6d8-42e8-adfe-58b6141471f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6817","nonce":49288782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6819","nonce":49288782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6823","nonce":49288782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":49288782},{"type":"v1","addr":"172.21.15.173:6821","nonce":49288782}]},"public_addr":"172.21.15.173:6817/49288782","cluster_addr":"172.21.15.173:6819/49288782","heartbeat_back_addr":"172.21.15.173:6823/49288782","heartbeat_front_addr":"172.21.15.173:6821/49288782","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:10:55.172572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:11:22.140314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:11:49.151682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:12:12.352626+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:12:37.457216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T06:13:02.197643+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6827/3380248129":"2024-02-11T06:14:33.003204+0000","172.21.15.89:0/632061317":"2024-02-11T06:08:38.557858+0000","172.21.15.89:6801/1397975713":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/2858441641":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/2878753730":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/2946515553":"2024-02-11T06:08:08.869122+0000","172.21.15.89:6800/3711070832":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/1754666852":"2024-02-11T06:09:42.089772+0000","172.21.15.89:6800/1397975713":"2024-02-11T06:08:38.557858+0000","172.21.15.89:0/478488956":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/3545264344":"2024-02-11T06:08:38.557858+0000","172.21.15.89:6800/1955086590":"2024-02-11T06:09:42.089772+0000","172.21.15.89:6801/3711070832":"2024-02-11T06:08:08.869122+0000","172.21.15.89:0/1015341098":"2024-02-11T06:09:42.089772+0000","172.21.15.89:0/644693661":"2024-02-11T06:09:42.089772+0000","172.21.15.89:6801/1955086590":"2024-02-11T06:09:42.089772+0000","172.21.15.89:6826/3380248129":"2024-02-11T06:14:33.003204+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T06:14:47.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.003+0000 7f7bab99d640 1 -- 172.21.15.89:0/432636092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7b90069210 msgr2=0x7f7b9006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.003+0000 7f7bab99d640 1 --2- 172.21.15.89:0/432636092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7b90069210 0x7f7b9006b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0009af0 tx=0x7f7ba0009cd0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.003+0000 7f7bab99d640 1 -- 172.21.15.89:0/432636092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ba40ff430 msgr2=0x7f7ba410d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.003+0000 7f7bab99d640 1 --2- 172.21.15.89:0/432636092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ba40ff430 0x7f7ba410d0a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7b940043d0 tx=0x7f7b94009da0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.003+0000 7f7bab99d640 1 -- 172.21.15.89:0/432636092 shutdown_connections 2024-02-10T06:14:47.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.003+0000 7f7bab99d640 1 --2- 172.21.15.89:0/432636092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f7b90069210 0x7f7b9006b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.003+0000 7f7bab99d640 1 --2- 172.21.15.89:0/432636092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ba40ff430 0x7f7ba410d0a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.003+0000 7f7bab99d640 1 --2- 172.21.15.89:0/432636092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ba4066350 0x7f7ba410cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.003+0000 7f7bab99d640 1 -- 172.21.15.89:0/432636092 >> 172.21.15.89:0/432636092 conn(0x7f7ba40f6940 msgr2=0x7f7ba40f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:47.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.003+0000 7f7bab99d640 1 -- 172.21.15.89:0/432636092 shutdown_connections 2024-02-10T06:14:47.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.003+0000 7f7bab99d640 1 -- 172.21.15.89:0/432636092 wait complete. 2024-02-10T06:14:47.016 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T06:11:52.416365+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-10T06:14:16.816453+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-10T06:14:17.466421+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-10T06:14:47.016 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-10T06:14:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.119+0000 7faa91225640 1 -- 172.21.15.89:0/1629346991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa8c0fc450 msgr2=0x7faa8c0fc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.119+0000 7faa91225640 1 --2- 172.21.15.89:0/1629346991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa8c0fc450 0x7faa8c0fc8c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7faa800048d0 tx=0x7faa80009da0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.119+0000 7faa91225640 1 -- 172.21.15.89:0/1629346991 shutdown_connections 2024-02-10T06:14:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.119+0000 7faa91225640 1 --2- 172.21.15.89:0/1629346991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa8c0fc450 0x7faa8c0fc8c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.119+0000 7faa91225640 1 --2- 172.21.15.89:0/1629346991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa8c0fb180 0x7faa8c0fb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.119+0000 7faa91225640 1 -- 172.21.15.89:0/1629346991 >> 172.21.15.89:0/1629346991 conn(0x7faa8c0f6960 msgr2=0x7faa8c0f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.119+0000 7faa91225640 1 -- 172.21.15.89:0/1629346991 shutdown_connections 2024-02-10T06:14:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.119+0000 7faa91225640 1 -- 172.21.15.89:0/1629346991 wait complete. 2024-02-10T06:14:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa91225640 1 Processor -- start 2024-02-10T06:14:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa91225640 1 -- start start 2024-02-10T06:14:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa91225640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa8c0fb180 0x7faa8c10cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa91225640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa8c0fc450 0x7faa8c10d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa91225640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c10d6e0 con 0x7faa8c0fc450 2024-02-10T06:14:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa91225640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c112130 con 0x7faa8c0fb180 2024-02-10T06:14:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa8a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa8c0fc450 0x7faa8c10d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa8a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa8c0fc450 0x7faa8c10d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34746/0 (socket says 172.21.15.89:34746) 2024-02-10T06:14:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa8a59c640 1 -- 172.21.15.89:0/1793679153 learned_addr learned my addr 172.21.15.89:0/1793679153 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa8ad9d640 1 --2- 172.21.15.89:0/1793679153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa8c0fb180 0x7faa8c10cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa8a59c640 1 -- 172.21.15.89:0/1793679153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa8c0fb180 msgr2=0x7faa8c10cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa8a59c640 1 --2- 172.21.15.89:0/1793679153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa8c0fb180 0x7faa8c10cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.120+0000 7faa8a59c640 1 -- 172.21.15.89:0/1793679153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa80005040 con 0x7faa8c0fc450 2024-02-10T06:14:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.121+0000 7faa8a59c640 1 --2- 172.21.15.89:0/1793679153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa8c0fc450 0x7faa8c10d090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7faa80007da0 tx=0x7faa80009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.121+0000 7faa737fe640 1 -- 172.21.15.89:0/1793679153 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa800094c0 con 0x7faa8c0fc450 2024-02-10T06:14:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.121+0000 7faa91225640 1 -- 172.21.15.89:0/1793679153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c112270 con 0x7faa8c0fc450 2024-02-10T06:14:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.121+0000 7faa737fe640 1 -- 172.21.15.89:0/1793679153 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa80009620 con 0x7faa8c0fc450 2024-02-10T06:14:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.121+0000 7faa737fe640 1 -- 172.21.15.89:0/1793679153 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa8003b620 con 0x7faa8c0fc450 2024-02-10T06:14:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.121+0000 7faa91225640 1 -- 172.21.15.89:0/1793679153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c112760 con 0x7faa8c0fc450 2024-02-10T06:14:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.122+0000 7faa91225640 1 -- 172.21.15.89:0/1793679153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c103cd0 con 0x7faa8c0fc450 2024-02-10T06:14:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.122+0000 7faa737fe640 1 -- 172.21.15.89:0/1793679153 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7faa8002f070 con 0x7faa8c0fc450 2024-02-10T06:14:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.122+0000 7faa737fe640 1 --2- 172.21.15.89:0/1793679153 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faa74069210 0x7faa7406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.122+0000 7faa8ad9d640 1 --2- 172.21.15.89:0/1793679153 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faa74069210 0x7faa7406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.122+0000 7faa737fe640 1 -- 172.21.15.89:0/1793679153 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7faa800a9850 con 0x7faa8c0fc450 2024-02-10T06:14:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.123+0000 7faa8ad9d640 1 --2- 172.21.15.89:0/1793679153 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faa74069210 0x7faa7406b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faa84004ce0 tx=0x7faa8400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.127+0000 7faa737fe640 1 -- 172.21.15.89:0/1793679153 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7faa80075ab0 con 0x7faa8c0fc450 2024-02-10T06:14:47.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.289+0000 7faa91225640 1 -- 172.21.15.89:0/1793679153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa8c1128f0 con 0x7faa8c0fc450 2024-02-10T06:14:47.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.290+0000 7faa737fe640 1 -- 172.21.15.89:0/1793679153 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa800790d0 con 0x7faa8c0fc450 2024-02-10T06:14:47.291 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-02-10T06:14:47.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.291+0000 7faa91225640 1 -- 172.21.15.89:0/1793679153 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faa74069210 msgr2=0x7faa7406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.291+0000 7faa91225640 1 --2- 172.21.15.89:0/1793679153 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faa74069210 0x7faa7406b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faa84004ce0 tx=0x7faa8400b040 comp rx=0 tx=0).stop 2024-02-10T06:14:47.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.291+0000 7faa91225640 1 -- 172.21.15.89:0/1793679153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa8c0fc450 msgr2=0x7faa8c10d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.291+0000 7faa91225640 1 --2- 172.21.15.89:0/1793679153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa8c0fc450 0x7faa8c10d090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7faa80007da0 tx=0x7faa80009da0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.292+0000 7faa91225640 1 -- 172.21.15.89:0/1793679153 shutdown_connections 2024-02-10T06:14:47.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.292+0000 7faa91225640 1 --2- 172.21.15.89:0/1793679153 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faa74069210 0x7faa7406b6c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.292+0000 7faa91225640 1 --2- 172.21.15.89:0/1793679153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa8c0fc450 0x7faa8c10d090 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.292+0000 7faa91225640 1 --2- 172.21.15.89:0/1793679153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa8c0fb180 0x7faa8c10cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.292+0000 7faa91225640 1 -- 172.21.15.89:0/1793679153 >> 172.21.15.89:0/1793679153 conn(0x7faa8c0f6960 msgr2=0x7faa8c065710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:47.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.292+0000 7faa91225640 1 -- 172.21.15.89:0/1793679153 shutdown_connections 2024-02-10T06:14:47.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.292+0000 7faa91225640 1 -- 172.21.15.89:0/1793679153 wait complete. 2024-02-10T06:14:47.302 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:46 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:47.302 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:46 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:47.303 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:46 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:47.303 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:46 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:47.303 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:46 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:47.303 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:46 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/184700042' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-10T06:14:47.303 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:46 smithi089 ceph-mon[29217]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T06:14:47.304 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-10T06:14:47.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:46 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:47.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:46 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:47.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:46 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:47.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:46 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:47.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:46 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:14:47.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:46 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/184700042' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-10T06:14:47.401 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:46 smithi173 ceph-mon[36251]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T06:14:47.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.403+0000 7ff55602e640 1 -- 172.21.15.89:0/3481421994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500ff450 msgr2=0x7ff550101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.403+0000 7ff55602e640 1 --2- 172.21.15.89:0/3481421994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500ff450 0x7ff550101830 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff5440048d0 tx=0x7ff544009da0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.403+0000 7ff55602e640 1 -- 172.21.15.89:0/3481421994 shutdown_connections 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.403+0000 7ff55602e640 1 --2- 172.21.15.89:0/3481421994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500ff450 0x7ff550101830 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.403+0000 7ff55602e640 1 --2- 172.21.15.89:0/3481421994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5500662f0 0x7ff5500fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.403+0000 7ff55602e640 1 -- 172.21.15.89:0/3481421994 >> 172.21.15.89:0/3481421994 conn(0x7ff5500f6960 msgr2=0x7ff5500f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.403+0000 7ff55602e640 1 -- 172.21.15.89:0/3481421994 shutdown_connections 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.403+0000 7ff55602e640 1 -- 172.21.15.89:0/3481421994 wait complete. 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.404+0000 7ff55602e640 1 Processor -- start 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.404+0000 7ff55602e640 1 -- start start 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.404+0000 7ff55602e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5500662f0 0x7ff55010cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.404+0000 7ff55602e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500ff450 0x7ff55010d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.404+0000 7ff55602e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55010d740 con 0x7ff5500ff450 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.404+0000 7ff55602e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff550112190 con 0x7ff5500662f0 2024-02-10T06:14:47.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.404+0000 7ff54effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500ff450 0x7ff55010d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.404+0000 7ff54effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500ff450 0x7ff55010d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34770/0 (socket says 172.21.15.89:34770) 2024-02-10T06:14:47.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.404+0000 7ff54effd640 1 -- 172.21.15.89:0/443303062 learned_addr learned my addr 172.21.15.89:0/443303062 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:47.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.404+0000 7ff54f7fe640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5500662f0 0x7ff55010cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.405+0000 7ff54effd640 1 -- 172.21.15.89:0/443303062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5500662f0 msgr2=0x7ff55010cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.405+0000 7ff54effd640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5500662f0 0x7ff55010cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.405+0000 7ff54effd640 1 -- 172.21.15.89:0/443303062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff544005040 con 0x7ff5500ff450 2024-02-10T06:14:47.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.405+0000 7ff54f7fe640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5500662f0 0x7ff55010cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:14:47.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.405+0000 7ff54effd640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500ff450 0x7ff55010d0f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff54402bda0 tx=0x7ff544009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:47.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.405+0000 7ff537fff640 1 -- 172.21.15.89:0/443303062 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5440094c0 con 0x7ff5500ff450 2024-02-10T06:14:47.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.405+0000 7ff537fff640 1 -- 172.21.15.89:0/443303062 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff544007cc0 con 0x7ff5500ff450 2024-02-10T06:14:47.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.405+0000 7ff55602e640 1 -- 172.21.15.89:0/443303062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5501122d0 con 0x7ff5500ff450 2024-02-10T06:14:47.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.405+0000 7ff537fff640 1 -- 172.21.15.89:0/443303062 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff54403b400 con 0x7ff5500ff450 2024-02-10T06:14:47.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.405+0000 7ff55602e640 1 -- 172.21.15.89:0/443303062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5501127c0 con 0x7ff5500ff450 2024-02-10T06:14:47.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.406+0000 7ff55602e640 1 -- 172.21.15.89:0/443303062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff550104e00 con 0x7ff5500ff450 2024-02-10T06:14:47.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.406+0000 7ff537fff640 1 -- 172.21.15.89:0/443303062 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff54402f070 con 0x7ff5500ff450 2024-02-10T06:14:47.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.407+0000 7ff537fff640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff538069210 0x7ff53806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.407+0000 7ff54f7fe640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff538069210 0x7ff53806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.407+0000 7ff537fff640 1 -- 172.21.15.89:0/443303062 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff5440a9770 con 0x7ff5500ff450 2024-02-10T06:14:47.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.407+0000 7ff54f7fe640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff538069210 0x7ff53806b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff550066050 tx=0x7ff54000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:47.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.411+0000 7ff537fff640 1 -- 172.21.15.89:0/443303062 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff5440759d0 con 0x7ff5500ff450 2024-02-10T06:14:47.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.576+0000 7ff55602e640 1 -- 172.21.15.89:0/443303062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7ff550112e20 con 0x7ff5500ff450 2024-02-10T06:14:47.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.576+0000 7ff537fff640 1 -- 172.21.15.89:0/443303062 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7ff544078ff0 con 0x7ff5500ff450 2024-02-10T06:14:47.580 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-02-10T06:14:47.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.578+0000 7ff55602e640 1 -- 172.21.15.89:0/443303062 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff538069210 msgr2=0x7ff53806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.578+0000 7ff55602e640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff538069210 0x7ff53806b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff550066050 tx=0x7ff54000b040 comp rx=0 tx=0).stop 2024-02-10T06:14:47.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.578+0000 7ff55602e640 1 -- 172.21.15.89:0/443303062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500ff450 msgr2=0x7ff55010d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.578+0000 7ff55602e640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500ff450 0x7ff55010d0f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff54402bda0 tx=0x7ff544009da0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.578+0000 7ff55602e640 1 -- 172.21.15.89:0/443303062 shutdown_connections 2024-02-10T06:14:47.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.578+0000 7ff55602e640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff538069210 0x7ff53806b6c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.578+0000 7ff55602e640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500ff450 0x7ff55010d0f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.578+0000 7ff55602e640 1 --2- 172.21.15.89:0/443303062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5500662f0 0x7ff55010cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.578+0000 7ff55602e640 1 -- 172.21.15.89:0/443303062 >> 172.21.15.89:0/443303062 conn(0x7ff5500f6960 msgr2=0x7ff5500f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:47.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.578+0000 7ff55602e640 1 -- 172.21.15.89:0/443303062 shutdown_connections 2024-02-10T06:14:47.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.578+0000 7ff55602e640 1 -- 172.21.15.89:0/443303062 wait complete. 2024-02-10T06:14:47.590 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-10T06:14:47.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.689+0000 7fc697e2e640 1 -- 172.21.15.89:0/2858024909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6900fd570 msgr2=0x7fc6900fd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.689+0000 7fc697e2e640 1 --2- 172.21.15.89:0/2858024909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6900fd570 0x7fc6900fd9e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc6800048d0 tx=0x7fc680009da0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.689+0000 7fc697e2e640 1 -- 172.21.15.89:0/2858024909 shutdown_connections 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.689+0000 7fc697e2e640 1 --2- 172.21.15.89:0/2858024909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6900fd570 0x7fc6900fd9e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.689+0000 7fc697e2e640 1 --2- 172.21.15.89:0/2858024909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6900fc2a0 0x7fc6900fc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.689+0000 7fc697e2e640 1 -- 172.21.15.89:0/2858024909 >> 172.21.15.89:0/2858024909 conn(0x7fc6900f7a40 msgr2=0x7fc6900f9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.689+0000 7fc697e2e640 1 -- 172.21.15.89:0/2858024909 shutdown_connections 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.689+0000 7fc697e2e640 1 -- 172.21.15.89:0/2858024909 wait complete. 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.689+0000 7fc697e2e640 1 Processor -- start 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.689+0000 7fc697e2e640 1 -- start start 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc697e2e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6900fc2a0 0x7fc690071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc697e2e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6900fd570 0x7fc69006f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc697e2e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69006fd30 con 0x7fc6900fc2a0 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc697e2e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69006fea0 con 0x7fc6900fd570 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc695bca640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6900fc2a0 0x7fc690071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc695bca640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6900fc2a0 0x7fc690071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34798/0 (socket says 172.21.15.89:34798) 2024-02-10T06:14:47.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc695bca640 1 -- 172.21.15.89:0/3592189513 learned_addr learned my addr 172.21.15.89:0/3592189513 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:47.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc695bca640 1 -- 172.21.15.89:0/3592189513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6900fd570 msgr2=0x7fc69006f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc6953c9640 1 --2- 172.21.15.89:0/3592189513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6900fd570 0x7fc69006f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc695bca640 1 --2- 172.21.15.89:0/3592189513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6900fd570 0x7fc69006f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc695bca640 1 -- 172.21.15.89:0/3592189513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc680005040 con 0x7fc6900fc2a0 2024-02-10T06:14:47.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc695bca640 1 --2- 172.21.15.89:0/3592189513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6900fc2a0 0x7fc690071170 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00b790 tx=0x7fc68c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:47.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc6867fc640 1 -- 172.21.15.89:0/3592189513 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc68c00c700 con 0x7fc6900fc2a0 2024-02-10T06:14:47.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc697e2e640 1 -- 172.21.15.89:0/3592189513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc690070180 con 0x7fc6900fc2a0 2024-02-10T06:14:47.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc6867fc640 1 -- 172.21.15.89:0/3592189513 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc68c00cd30 con 0x7fc6900fc2a0 2024-02-10T06:14:47.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.690+0000 7fc6867fc640 1 -- 172.21.15.89:0/3592189513 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc68c013370 con 0x7fc6900fc2a0 2024-02-10T06:14:47.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.691+0000 7fc697e2e640 1 -- 172.21.15.89:0/3592189513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc690114cd0 con 0x7fc6900fc2a0 2024-02-10T06:14:47.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.691+0000 7fc697e2e640 1 -- 172.21.15.89:0/3592189513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc690104e70 con 0x7fc6900fc2a0 2024-02-10T06:14:47.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.692+0000 7fc6867fc640 1 -- 172.21.15.89:0/3592189513 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc68c012070 con 0x7fc6900fc2a0 2024-02-10T06:14:47.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.692+0000 7fc6867fc640 1 --2- 172.21.15.89:0/3592189513 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc67c069210 0x7fc67c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.692+0000 7fc6867fc640 1 -- 172.21.15.89:0/3592189513 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc68c086e70 con 0x7fc6900fc2a0 2024-02-10T06:14:47.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.692+0000 7fc6953c9640 1 --2- 172.21.15.89:0/3592189513 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc67c069210 0x7fc67c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.693+0000 7fc6953c9640 1 --2- 172.21.15.89:0/3592189513 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc67c069210 0x7fc67c06b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc680003080 tx=0x7fc680009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:47.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.697+0000 7fc6867fc640 1 -- 172.21.15.89:0/3592189513 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc68c0530d0 con 0x7fc6900fc2a0 2024-02-10T06:14:47.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.859+0000 7fc697e2e640 1 -- 172.21.15.89:0/3592189513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fc690105000 con 0x7fc6900fc2a0 2024-02-10T06:14:47.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.860+0000 7fc6867fc640 1 -- 172.21.15.89:0/3592189513 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc690105000 con 0x7fc6900fc2a0 2024-02-10T06:14:47.861 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-02-10T06:14:47.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.861+0000 7fc697e2e640 1 -- 172.21.15.89:0/3592189513 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc67c069210 msgr2=0x7fc67c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.861+0000 7fc697e2e640 1 --2- 172.21.15.89:0/3592189513 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc67c069210 0x7fc67c06b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc680003080 tx=0x7fc680009da0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.861+0000 7fc697e2e640 1 -- 172.21.15.89:0/3592189513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6900fc2a0 msgr2=0x7fc690071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.861+0000 7fc697e2e640 1 --2- 172.21.15.89:0/3592189513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6900fc2a0 0x7fc690071170 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00b790 tx=0x7fc68c0091d0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.861+0000 7fc697e2e640 1 -- 172.21.15.89:0/3592189513 shutdown_connections 2024-02-10T06:14:47.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.861+0000 7fc697e2e640 1 --2- 172.21.15.89:0/3592189513 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fc67c069210 0x7fc67c06b6c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.861+0000 7fc697e2e640 1 --2- 172.21.15.89:0/3592189513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6900fd570 0x7fc69006f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.861+0000 7fc697e2e640 1 --2- 172.21.15.89:0/3592189513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6900fc2a0 0x7fc690071170 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.861+0000 7fc697e2e640 1 -- 172.21.15.89:0/3592189513 >> 172.21.15.89:0/3592189513 conn(0x7fc6900f7a40 msgr2=0x7fc6901007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:47.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.862+0000 7fc697e2e640 1 -- 172.21.15.89:0/3592189513 shutdown_connections 2024-02-10T06:14:47.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.862+0000 7fc697e2e640 1 -- 172.21.15.89:0/3592189513 wait complete. 2024-02-10T06:14:47.874 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-10T06:14:47.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 -- 172.21.15.89:0/2709490066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a700fb170 msgr2=0x7f4a700fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 --2- 172.21.15.89:0/2709490066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a700fb170 0x7f4a700fb580 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4a600048d0 tx=0x7f4a60009da0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 -- 172.21.15.89:0/2709490066 shutdown_connections 2024-02-10T06:14:47.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 --2- 172.21.15.89:0/2709490066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a700fc440 0x7f4a700fc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 --2- 172.21.15.89:0/2709490066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a700fb170 0x7f4a700fb580 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 -- 172.21.15.89:0/2709490066 >> 172.21.15.89:0/2709490066 conn(0x7f4a700f6910 msgr2=0x7f4a700f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:47.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 -- 172.21.15.89:0/2709490066 shutdown_connections 2024-02-10T06:14:47.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 -- 172.21.15.89:0/2709490066 wait complete. 2024-02-10T06:14:47.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 Processor -- start 2024-02-10T06:14:47.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 -- start start 2024-02-10T06:14:47.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a700fb170 0x7f4a7010baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a700fc440 0x7f4a7010bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7010c6c0 con 0x7f4a700fb170 2024-02-10T06:14:47.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a75065640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a70111080 con 0x7f4a700fc440 2024-02-10T06:14:47.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a6ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a700fb170 0x7f4a7010baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a6ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a700fb170 0x7f4a7010baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34818/0 (socket says 172.21.15.89:34818) 2024-02-10T06:14:47.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.970+0000 7f4a6ed9d640 1 -- 172.21.15.89:0/2720920824 learned_addr learned my addr 172.21.15.89:0/2720920824 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:47.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.971+0000 7f4a67fff640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a700fc440 0x7f4a7010bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.971+0000 7f4a6ed9d640 1 -- 172.21.15.89:0/2720920824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a700fc440 msgr2=0x7f4a7010bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:47.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.971+0000 7f4a6ed9d640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a700fc440 0x7f4a7010bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:47.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.971+0000 7f4a6ed9d640 1 -- 172.21.15.89:0/2720920824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a60005040 con 0x7f4a700fb170 2024-02-10T06:14:47.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.971+0000 7f4a67fff640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a700fc440 0x7f4a7010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:14:47.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.971+0000 7f4a6ed9d640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a700fb170 0x7f4a7010baa0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4a60007d20 tx=0x7f4a60009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:47.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.971+0000 7f4a677fe640 1 -- 172.21.15.89:0/2720920824 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a60009590 con 0x7f4a700fb170 2024-02-10T06:14:47.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.971+0000 7f4a677fe640 1 -- 172.21.15.89:0/2720920824 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a600096f0 con 0x7f4a700fb170 2024-02-10T06:14:47.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.971+0000 7f4a75065640 1 -- 172.21.15.89:0/2720920824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a70111280 con 0x7f4a700fb170 2024-02-10T06:14:47.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.971+0000 7f4a677fe640 1 -- 172.21.15.89:0/2720920824 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a6003b6d0 con 0x7f4a700fb170 2024-02-10T06:14:47.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.971+0000 7f4a75065640 1 -- 172.21.15.89:0/2720920824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a701117a0 con 0x7f4a700fb170 2024-02-10T06:14:47.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.972+0000 7f4a75065640 1 -- 172.21.15.89:0/2720920824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a70103d40 con 0x7f4a700fb170 2024-02-10T06:14:47.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.972+0000 7f4a677fe640 1 -- 172.21.15.89:0/2720920824 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4a6002f070 con 0x7f4a700fb170 2024-02-10T06:14:47.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.973+0000 7f4a677fe640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4a5c069210 0x7f4a5c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:47.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.973+0000 7f4a677fe640 1 -- 172.21.15.89:0/2720920824 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4a600a9940 con 0x7f4a700fb170 2024-02-10T06:14:47.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.973+0000 7f4a67fff640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4a5c069210 0x7f4a5c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:47.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.973+0000 7f4a67fff640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4a5c069210 0x7f4a5c06b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4a58006fd0 tx=0x7f4a58008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:47.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:47.977+0000 7f4a677fe640 1 -- 172.21.15.89:0/2720920824 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4a60075ba0 con 0x7f4a700fb170 2024-02-10T06:14:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.177+0000 7f4a75065640 1 -- 172.21.15.89:0/2720920824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a70111930 con 0x7f4a700fb170 2024-02-10T06:14:48.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.178+0000 7f4a677fe640 1 -- 172.21.15.89:0/2720920824 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4702 (secure 0 0 0) 0x7f4a600791c0 con 0x7f4a700fb170 2024-02-10T06:14:48.179 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:14:48.179 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24243,"name":"cephfs.smithi173.cjkmdf","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.173:6827/2798078299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":2798078299},{"type":"v1","addr":"172.21.15.173:6827","nonce":2798078299}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24251,"name":"cephfs.smithi089.mgkvmj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6827/3574254294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3574254294},{"type":"v1","addr":"172.21.15.89:6827","nonce":3574254294}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-10T06:14:18.236041+0000","modified":"2024-02-10T06:14:36.018945+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14476":{"gid":14476,"name":"cephfs.smithi089.evydlq","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.89:6829/3386828748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":3386828748},{"type":"v1","addr":"172.21.15.89:6829","nonce":3386828748}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24235":{"gid":24235,"name":"cephfs.smithi173.tumnkz","rank":0,"incarnation":11,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6825/3924282139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3924282139},{"type":"v1","addr":"172.21.15.173:6825","nonce":3924282139}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-02-10T06:14:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 15 2024-02-10T06:14:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.180+0000 7f4a75065640 1 -- 172.21.15.89:0/2720920824 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4a5c069210 msgr2=0x7f4a5c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.180+0000 7f4a75065640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4a5c069210 0x7f4a5c06b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4a58006fd0 tx=0x7f4a58008040 comp rx=0 tx=0).stop 2024-02-10T06:14:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.180+0000 7f4a75065640 1 -- 172.21.15.89:0/2720920824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a700fb170 msgr2=0x7f4a7010baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.180+0000 7f4a75065640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a700fb170 0x7f4a7010baa0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4a60007d20 tx=0x7f4a60009da0 comp rx=0 tx=0).stop 2024-02-10T06:14:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.180+0000 7f4a75065640 1 -- 172.21.15.89:0/2720920824 shutdown_connections 2024-02-10T06:14:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.180+0000 7f4a75065640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4a5c069210 0x7f4a5c06b6c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.180+0000 7f4a75065640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a700fc440 0x7f4a7010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.180+0000 7f4a75065640 1 --2- 172.21.15.89:0/2720920824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a700fb170 0x7f4a7010baa0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.180+0000 7f4a75065640 1 -- 172.21.15.89:0/2720920824 >> 172.21.15.89:0/2720920824 conn(0x7f4a700f6910 msgr2=0x7f4a700ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.180+0000 7f4a75065640 1 -- 172.21.15.89:0/2720920824 shutdown_connections 2024-02-10T06:14:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:48.180+0000 7f4a75065640 1 -- 172.21.15.89:0/2720920824 wait complete. 2024-02-10T06:14:48.190 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:47 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/432636092' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:14:48.190 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:47 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1793679153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T06:14:48.190 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:47 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/443303062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-10T06:14:48.190 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:47 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3592189513' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-10T06:14:48.191 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 50} 2024-02-10T06:14:48.191 INFO:teuthology.run_tasks:Running task kclient... 2024-02-10T06:14:48.218 INFO:tasks.kclient:Mounting kernel clients... 2024-02-10T06:14:48.219 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-02-10T06:14:48.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:48.219 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-02-10T06:14:48.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:48.234 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2024-02-10T06:14:48.302 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:47 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/432636092' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T06:14:48.302 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:47 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/1793679153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T06:14:48.302 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:47 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/443303062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-10T06:14:48.302 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:47 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/3592189513' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-10T06:14:48.303 INFO:teuthology.orchestra.run.smithi089.stderr:Cannot find device "ceph-brx" 2024-02-10T06:14:48.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:14:48.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:48.305 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-02-10T06:14:48.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:48.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link delete ceph-brx 2024-02-10T06:14:48.399 INFO:teuthology.orchestra.run.smithi173.stderr:Cannot find device "ceph-brx" 2024-02-10T06:14:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:14:48.400 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-02-10T06:14:48.400 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-10T06:14:48.400 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-10T06:14:48.400 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph fs ls 2024-02-10T06:14:48.651 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:49.075 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:49 smithi089 ceph-mon[29217]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T06:14:49.076 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:49 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2720920824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-10T06:14:49.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:49 smithi173 ceph-mon[36251]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-10T06:14:49.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:49 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2720920824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-10T06:14:50.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.238+0000 7faae8b25700 1 -- 172.21.15.89:0/2773005663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae4101ee0 msgr2=0x7faae41022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:50.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.238+0000 7faae8b25700 1 --2- 172.21.15.89:0/2773005663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae4101ee0 0x7faae41022b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faad8004940 tx=0x7faad8009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:50.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.238+0000 7faae8b25700 1 -- 172.21.15.89:0/2773005663 shutdown_connections 2024-02-10T06:14:50.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.238+0000 7faae8b25700 1 --2- 172.21.15.89:0/2773005663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae4102880 0x7faae40fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:50.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.238+0000 7faae8b25700 1 --2- 172.21.15.89:0/2773005663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae4101ee0 0x7faae41022b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:50.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.238+0000 7faae8b25700 1 -- 172.21.15.89:0/2773005663 >> 172.21.15.89:0/2773005663 conn(0x7faae4071560 msgr2=0x7faae4071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:50.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.238+0000 7faae8b25700 1 -- 172.21.15.89:0/2773005663 shutdown_connections 2024-02-10T06:14:50.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.238+0000 7faae8b25700 1 -- 172.21.15.89:0/2773005663 wait complete. 2024-02-10T06:14:50.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae8b25700 1 Processor -- start 2024-02-10T06:14:50.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae8b25700 1 -- start start 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae8b25700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae4101ee0 0x7faae4112940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae8b25700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae4102880 0x7faae4112e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae8b25700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae4113560 con 0x7faae4102880 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae8b25700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae410c800 con 0x7faae4101ee0 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae1d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae4102880 0x7faae4112e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae1d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae4102880 0x7faae4112e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34654/0 (socket says 172.21.15.89:34654) 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae1d9b700 1 -- 172.21.15.89:0/323259586 learned_addr learned my addr 172.21.15.89:0/323259586 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae259c700 1 --2- 172.21.15.89:0/323259586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae4101ee0 0x7faae4112940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae1d9b700 1 -- 172.21.15.89:0/323259586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae4101ee0 msgr2=0x7faae4112940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae1d9b700 1 --2- 172.21.15.89:0/323259586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae4101ee0 0x7faae4112940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.239+0000 7faae1d9b700 1 -- 172.21.15.89:0/323259586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad8005040 con 0x7faae4102880 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.240+0000 7faae1d9b700 1 --2- 172.21.15.89:0/323259586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae4102880 0x7faae4112e80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faad400b790 tx=0x7faad4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.240+0000 7faad2ffd700 1 -- 172.21.15.89:0/323259586 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faad400f6f0 con 0x7faae4102880 2024-02-10T06:14:50.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.240+0000 7faad2ffd700 1 -- 172.21.15.89:0/323259586 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faad400fd20 con 0x7faae4102880 2024-02-10T06:14:50.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.240+0000 7faae8b25700 1 -- 172.21.15.89:0/323259586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae410cae0 con 0x7faae4102880 2024-02-10T06:14:50.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.240+0000 7faad2ffd700 1 -- 172.21.15.89:0/323259586 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faad40103a0 con 0x7faae4102880 2024-02-10T06:14:50.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.240+0000 7faae8b25700 1 -- 172.21.15.89:0/323259586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae410d020 con 0x7faae4102880 2024-02-10T06:14:50.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.241+0000 7faae8b25700 1 -- 172.21.15.89:0/323259586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae4073080 con 0x7faae4102880 2024-02-10T06:14:50.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.241+0000 7faad2ffd700 1 -- 172.21.15.89:0/323259586 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7faad400f850 con 0x7faae4102880 2024-02-10T06:14:50.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.242+0000 7faad2ffd700 1 --2- 172.21.15.89:0/323259586 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faacc0692a0 0x7faacc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:50.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.242+0000 7faae259c700 1 --2- 172.21.15.89:0/323259586 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faacc0692a0 0x7faacc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:50.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.242+0000 7faad2ffd700 1 -- 172.21.15.89:0/323259586 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7faad4086f90 con 0x7faae4102880 2024-02-10T06:14:50.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.242+0000 7faae259c700 1 --2- 172.21.15.89:0/323259586 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faacc0692a0 0x7faacc06b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faad8004f90 tx=0x7faad8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:50.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.246+0000 7faad2ffd700 1 -- 172.21.15.89:0/323259586 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7faad4056d00 con 0x7faae4102880 2024-02-10T06:14:50.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.477+0000 7faae8b25700 1 -- 172.21.15.89:0/323259586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faae410df20 con 0x7faae4102880 2024-02-10T06:14:50.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.477+0000 7faad2ffd700 1 -- 172.21.15.89:0/323259586 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7faad4056890 con 0x7faae4102880 2024-02-10T06:14:50.478 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-10T06:14:50.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.479+0000 7faae8b25700 1 -- 172.21.15.89:0/323259586 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faacc0692a0 msgr2=0x7faacc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:50.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.479+0000 7faae8b25700 1 --2- 172.21.15.89:0/323259586 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faacc0692a0 0x7faacc06b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faad8004f90 tx=0x7faad8009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:50.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.479+0000 7faae8b25700 1 -- 172.21.15.89:0/323259586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae4102880 msgr2=0x7faae4112e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:50.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.479+0000 7faae8b25700 1 --2- 172.21.15.89:0/323259586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae4102880 0x7faae4112e80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faad400b790 tx=0x7faad4009220 comp rx=0 tx=0).stop 2024-02-10T06:14:50.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.479+0000 7faae8b25700 1 -- 172.21.15.89:0/323259586 shutdown_connections 2024-02-10T06:14:50.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.479+0000 7faae8b25700 1 --2- 172.21.15.89:0/323259586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faae4101ee0 0x7faae4112940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:50.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.479+0000 7faae8b25700 1 --2- 172.21.15.89:0/323259586 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7faacc0692a0 0x7faacc06b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:50.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.479+0000 7faae8b25700 1 --2- 172.21.15.89:0/323259586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae4102880 0x7faae4112e80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:50.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.479+0000 7faae8b25700 1 -- 172.21.15.89:0/323259586 >> 172.21.15.89:0/323259586 conn(0x7faae4071560 msgr2=0x7faae40fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:50.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.479+0000 7faae8b25700 1 -- 172.21.15.89:0/323259586 shutdown_connections 2024-02-10T06:14:50.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:50.480+0000 7faae8b25700 1 -- 172.21.15.89:0/323259586 wait complete. 2024-02-10T06:14:51.219 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-10T06:14:51.219 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-10T06:14:51.219 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-02-10T06:14:51.219 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-10T06:14:51.219 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-10T06:14:51.219 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-10T06:14:51.219 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-10T06:14:51.219 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-10T06:14:51.219 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-10T06:14:51.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:51.220 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-02-10T06:14:51.236 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-10T06:14:51.236 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-10T06:14:51.236 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-02-10T06:14:51.236 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:14:51.236 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-02-10T06:14:51.236 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:14:51.236 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:14:51.236 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42286sec preferred_lft 42286sec 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2024-02-10T06:14:51.237 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:14:51.238 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-10T06:14:51.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:51.238 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-02-10T06:14:51.238 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-02-10T06:14:51.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add name ceph-brx type bridge 2024-02-10T06:14:51.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr flush dev ceph-brx 2024-02-10T06:14:51.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set ceph-brx up 2024-02-10T06:14:51.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-10T06:14:51.238 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-02-10T06:14:51.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:51.412 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-10T06:14:51.439 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-02-10T06:14:51.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:51.441 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-02-10T06:14:51.510 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-02-10T06:14:51.511 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-10T06:14:51.511 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-10T06:14:51.511 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-10T06:14:51.511 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-10T06:14:51.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:51.511 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-02-10T06:14:51.511 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-02-10T06:14:51.511 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-10T06:14:51.512 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-10T06:14:51.512 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-10T06:14:51.512 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-02-10T06:14:51.575 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:51 smithi089 ceph-mon[29217]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:14:51.576 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:51 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/323259586' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-10T06:14:51.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:51 smithi173 ceph-mon[36251]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:14:51.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:51 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/323259586' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-10T06:14:51.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:51.796 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-02-10T06:14:51.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:51.852 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2024-02-10T06:14:51.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:51.907 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-02-10T06:14:51.907 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-02-10T06:14:51.907 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-10T06:14:51.907 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-10T06:14:51.907 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-02-10T06:14:51.989 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-10T06:14:51.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:51.989 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-02-10T06:14:51.989 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-02-10T06:14:51.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-10T06:14:51.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-10T06:14:51.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-10T06:14:51.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-10T06:14:51.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-10T06:14:51.990 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-02-10T06:14:52.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:52.127 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2024-02-10T06:14:52.128 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-02-10T06:14:52.128 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-02-10T06:14:52.128 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.0 master ceph-brx 2024-02-10T06:14:52.128 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-02-10T06:14:52.169 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-10T06:14:52.169 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-10T06:14:52.225 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-10T06:14:52.225 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-10T06:14:52.225 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-10T06:14:52.280 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-10T06:14:52.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:52.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-10T06:14:52.372 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-10T06:14:52.373 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-10T06:14:52.373 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2024-02-10T06:14:52.472 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-10T06:14:52.472 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-10T06:14:52.473 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2024-02-10T06:14:52.473 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.173:3300" 2024-02-10T06:14:52.473 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-10T06:14:52.478 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-10T06:14:52.478 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-10T06:14:52.478 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2024-02-10T06:14:52.478 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.173:3300" 2024-02-10T06:14:52.478 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-10T06:14:52.478 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.173:6789" 2024-02-10T06:14:52.478 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.173:6789:/ 2024-02-10T06:14:52.478 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=84e5b1ba-c7da-11ee-95b8-87774f69a715" will pass to kernel 2024-02-10T06:14:52.481 INFO:teuthology.orchestra.run.smithi089.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-10T06:14:52.481 INFO:teuthology.orchestra.run.smithi089.stdout: You just mounted a file system that supports labels which does not 2024-02-10T06:14:52.481 INFO:teuthology.orchestra.run.smithi089.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-10T06:14:52.481 INFO:teuthology.orchestra.run.smithi089.stdout: applications will generate AVC messages and not be allowed access to 2024-02-10T06:14:52.481 INFO:teuthology.orchestra.run.smithi089.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-10T06:14:52.482 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-10T06:14:52.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:52.483 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-10T06:14:52.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:52.511 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:14:52.573 INFO:teuthology.orchestra.run.smithi089.stdout:client14524 2024-02-10T06:14:52.573 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:14:52.573 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:14:52.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:52.630 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:14:52.693 INFO:teuthology.orchestra.run.smithi089.stdout:client14524 2024-02-10T06:14:52.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:52.693 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:14:52.756 INFO:teuthology.orchestra.run.smithi089.stdout:client14524 2024-02-10T06:14:52.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:52.757 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715.client14524/status') 2024-02-10T06:14:52.828 INFO:teuthology.orchestra.run.smithi089.stdout:instance: client.14524 (3)192.168.0.1:0/1330752345 2024-02-10T06:14:52.828 INFO:teuthology.orchestra.run.smithi089.stdout:blocklisted: false 2024-02-10T06:14:52.829 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records in 2024-02-10T06:14:52.829 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records out 2024-02-10T06:14:52.829 INFO:teuthology.orchestra.run.smithi089.stderr:70 bytes copied, 2.0273e-05 s, 3.5 MB/s 2024-02-10T06:14:52.830 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-02-10T06:14:52.830 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-10T06:14:52.830 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-10T06:14:52.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -- ceph fs ls 2024-02-10T06:14:53.130 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:53.489 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:53 smithi089 ceph-mon[29217]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:14:53.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:53 smithi173 ceph-mon[36251]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.736+0000 7ff46d82a700 1 -- 172.21.15.89:0/2927446683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff468103140 msgr2=0x7ff468103510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.736+0000 7ff46d82a700 1 --2- 172.21.15.89:0/2927446683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff468103140 0x7ff468103510 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff458004940 tx=0x7ff458009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.736+0000 7ff46d82a700 1 -- 172.21.15.89:0/2927446683 shutdown_connections 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.736+0000 7ff46d82a700 1 --2- 172.21.15.89:0/2927446683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680fb570 0x7ff4680fb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.736+0000 7ff46d82a700 1 --2- 172.21.15.89:0/2927446683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff468103140 0x7ff468103510 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.736+0000 7ff46d82a700 1 -- 172.21.15.89:0/2927446683 >> 172.21.15.89:0/2927446683 conn(0x7ff468071560 msgr2=0x7ff468071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.736+0000 7ff46d82a700 1 -- 172.21.15.89:0/2927446683 shutdown_connections 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.736+0000 7ff46d82a700 1 -- 172.21.15.89:0/2927446683 wait complete. 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.736+0000 7ff46d82a700 1 Processor -- start 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff46d82a700 1 -- start start 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff46d82a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680fb570 0x7ff468113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff46d82a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff468103140 0x7ff468114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff46d82a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4681146a0 con 0x7ff468103140 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff46d82a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46810d9d0 con 0x7ff4680fb570 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff4667fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff468103140 0x7ff468114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff4667fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff468103140 0x7ff468114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34668/0 (socket says 172.21.15.89:34668) 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff4667fc700 1 -- 172.21.15.89:0/3925082673 learned_addr learned my addr 172.21.15.89:0/3925082673 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff466ffd700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680fb570 0x7ff468113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff4667fc700 1 -- 172.21.15.89:0/3925082673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680fb570 msgr2=0x7ff468113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff4667fc700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680fb570 0x7ff468113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff4667fc700 1 -- 172.21.15.89:0/3925082673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff458005040 con 0x7ff468103140 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff466ffd700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680fb570 0x7ff468113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.737+0000 7ff4667fc700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff468103140 0x7ff468114050 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff45c010c40 tx=0x7ff45c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.738+0000 7ff44f7fe700 1 -- 172.21.15.89:0/3925082673 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff45c00ab60 con 0x7ff468103140 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.738+0000 7ff44f7fe700 1 -- 172.21.15.89:0/3925082673 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff45c00acc0 con 0x7ff468103140 2024-02-10T06:14:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.738+0000 7ff46d82a700 1 -- 172.21.15.89:0/3925082673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46810dc50 con 0x7ff468103140 2024-02-10T06:14:54.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.738+0000 7ff44f7fe700 1 -- 172.21.15.89:0/3925082673 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff45c017770 con 0x7ff468103140 2024-02-10T06:14:54.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.738+0000 7ff46d82a700 1 -- 172.21.15.89:0/3925082673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff46810e190 con 0x7ff468103140 2024-02-10T06:14:54.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.739+0000 7ff46d82a700 1 -- 172.21.15.89:0/3925082673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff468100310 con 0x7ff468103140 2024-02-10T06:14:54.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.739+0000 7ff44f7fe700 1 -- 172.21.15.89:0/3925082673 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff45c018b40 con 0x7ff468103140 2024-02-10T06:14:54.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.739+0000 7ff44f7fe700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4500692a0 0x7ff45006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:54.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.740+0000 7ff44f7fe700 1 -- 172.21.15.89:0/3925082673 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff45c08b370 con 0x7ff468103140 2024-02-10T06:14:54.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.740+0000 7ff466ffd700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4500692a0 0x7ff45006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:54.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.740+0000 7ff466ffd700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4500692a0 0x7ff45006b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff458004560 tx=0x7ff458009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:54.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.744+0000 7ff44f7fe700 1 -- 172.21.15.89:0/3925082673 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff45c05b0e0 con 0x7ff468103140 2024-02-10T06:14:54.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.968+0000 7ff46d82a700 1 -- 172.21.15.89:0/3925082673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff46810a1b0 con 0x7ff468103140 2024-02-10T06:14:54.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.968+0000 7ff44f7fe700 1 -- 172.21.15.89:0/3925082673 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7ff45c05ac70 con 0x7ff468103140 2024-02-10T06:14:54.969 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-10T06:14:54.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.970+0000 7ff46d82a700 1 -- 172.21.15.89:0/3925082673 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4500692a0 msgr2=0x7ff45006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:54.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.970+0000 7ff46d82a700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4500692a0 0x7ff45006b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff458004560 tx=0x7ff458009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:54.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.970+0000 7ff46d82a700 1 -- 172.21.15.89:0/3925082673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff468103140 msgr2=0x7ff468114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:54.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.970+0000 7ff46d82a700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff468103140 0x7ff468114050 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff45c010c40 tx=0x7ff45c004470 comp rx=0 tx=0).stop 2024-02-10T06:14:54.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.970+0000 7ff46d82a700 1 -- 172.21.15.89:0/3925082673 shutdown_connections 2024-02-10T06:14:54.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.970+0000 7ff46d82a700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4680fb570 0x7ff468113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:54.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.970+0000 7ff46d82a700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4500692a0 0x7ff45006b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:54.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.970+0000 7ff46d82a700 1 --2- 172.21.15.89:0/3925082673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff468103140 0x7ff468114050 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:54.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.970+0000 7ff46d82a700 1 -- 172.21.15.89:0/3925082673 >> 172.21.15.89:0/3925082673 conn(0x7ff468071560 msgr2=0x7ff4680fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:54.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.970+0000 7ff46d82a700 1 -- 172.21.15.89:0/3925082673 shutdown_connections 2024-02-10T06:14:54.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:54.970+0000 7ff46d82a700 1 -- 172.21.15.89:0/3925082673 wait complete. 2024-02-10T06:14:55.274 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:55 smithi089 ceph-mon[29217]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-02-10T06:14:55.274 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:55 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3925082673' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-10T06:14:55.600 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-10T06:14:55.601 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-10T06:14:55.601 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi173.front.sepia.ceph.com 2024-02-10T06:14:55.601 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-10T06:14:55.601 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-10T06:14:55.601 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-10T06:14:55.601 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-10T06:14:55.601 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-10T06:14:55.601 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-10T06:14:55.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:55.601 DEBUG:teuthology.orchestra.run.smithi173:> ip addr 2024-02-10T06:14:55.621 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-10T06:14:55.621 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-10T06:14:55.621 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-02-10T06:14:55.621 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:14:55.621 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-02-10T06:14:55.621 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:14:55.621 INFO:teuthology.orchestra.run.smithi173.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6a brd ff:ff:ff:ff:ff:ff 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f0 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6b brd ff:ff:ff:ff:ff:ff 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f1 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:58 brd ff:ff:ff:ff:ff:ff 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:59 brd ff:ff:ff:ff:ff:ff 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout: inet 172.21.15.173/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft 42300sec preferred_lft 42300sec 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::ec4:7aff:fe88:7259/64 scope link 2024-02-10T06:14:55.622 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-02-10T06:14:55.623 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-10T06:14:55.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:55.623 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-02-10T06:14:55.623 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-02-10T06:14:55.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link add name ceph-brx type bridge 2024-02-10T06:14:55.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip addr flush dev ceph-brx 2024-02-10T06:14:55.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set ceph-brx up 2024-02-10T06:14:55.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-10T06:14:55.623 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-02-10T06:14:55.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:55 smithi173 ceph-mon[36251]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-02-10T06:14:55.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:55 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/3925082673' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-10T06:14:55.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:55.783 DEBUG:teuthology.orchestra.run.smithi173:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-10T06:14:55.811 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-02-10T06:14:55.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:55.813 DEBUG:teuthology.orchestra.run.smithi173:> route 2024-02-10T06:14:55.889 INFO:teuthology.orchestra.run.smithi173.stdout:Kernel IP routing table 2024-02-10T06:14:55.889 INFO:teuthology.orchestra.run.smithi173.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-10T06:14:55.889 INFO:teuthology.orchestra.run.smithi173.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-10T06:14:55.889 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-10T06:14:55.889 INFO:teuthology.orchestra.run.smithi173.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-10T06:14:55.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:55.889 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-02-10T06:14:55.890 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-02-10T06:14:55.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-10T06:14:55.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-10T06:14:55.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-10T06:14:55.890 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-02-10T06:14:56.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:56.160 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-02-10T06:14:56.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:56.176 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list-id 2024-02-10T06:14:56.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:56.233 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-02-10T06:14:56.233 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-02-10T06:14:56.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-10T06:14:56.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-10T06:14:56.233 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-02-10T06:14:56.319 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-10T06:14:56.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:56.320 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-02-10T06:14:56.320 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-02-10T06:14:56.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-10T06:14:56.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-10T06:14:56.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-10T06:14:56.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-10T06:14:56.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-10T06:14:56.320 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-02-10T06:14:56.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:56.457 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-02-10T06:14:56.457 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-02-10T06:14:56.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set brx.0 up 2024-02-10T06:14:56.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set dev brx.0 master ceph-brx 2024-02-10T06:14:56.457 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-02-10T06:14:56.499 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-10T06:14:56.499 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-10T06:14:56.561 INFO:teuthology.orchestra.run.smithi173.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-10T06:14:56.562 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-10T06:14:56.562 DEBUG:teuthology.orchestra.run.smithi173:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-10T06:14:56.627 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-10T06:14:56.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:56.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-10T06:14:56.743 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-10T06:14:56.743 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-10T06:14:56.743 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-02-10T06:14:56.844 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-10T06:14:56.845 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-10T06:14:56.845 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-02-10T06:14:56.845 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.173:3300" 2024-02-10T06:14:56.845 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-10T06:14:56.848 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-10T06:14:56.848 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-10T06:14:56.848 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-02-10T06:14:56.848 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.173:3300" 2024-02-10T06:14:56.848 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-10T06:14:56.848 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.173:6789" 2024-02-10T06:14:56.848 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.173:6789:/ 2024-02-10T06:14:56.849 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=84e5b1ba-c7da-11ee-95b8-87774f69a715" will pass to kernel 2024-02-10T06:14:56.856 INFO:teuthology.orchestra.run.smithi173.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-02-10T06:14:56.856 INFO:teuthology.orchestra.run.smithi173.stdout: You just mounted a file system that supports labels which does not 2024-02-10T06:14:56.856 INFO:teuthology.orchestra.run.smithi173.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-10T06:14:56.856 INFO:teuthology.orchestra.run.smithi173.stdout: applications will generate AVC messages and not be allowed access to 2024-02-10T06:14:56.856 INFO:teuthology.orchestra.run.smithi173.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-10T06:14:56.858 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-10T06:14:56.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:56.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-10T06:14:56.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:56.890 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:14:56.949 INFO:teuthology.orchestra.run.smithi173.stdout:client24271 2024-02-10T06:14:56.949 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:14:56.949 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:14:57.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:57.010 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:14:57.070 INFO:teuthology.orchestra.run.smithi173.stdout:client24271 2024-02-10T06:14:57.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:57.070 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-10T06:14:57.128 INFO:teuthology.orchestra.run.smithi173.stdout:client24271 2024-02-10T06:14:57.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:14:57.129 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715.client24271/status') 2024-02-10T06:14:57.197 INFO:teuthology.orchestra.run.smithi173.stdout:instance: client.24271 (3)192.168.0.1:0/1129789111 2024-02-10T06:14:57.197 INFO:teuthology.orchestra.run.smithi173.stdout:blocklisted: false 2024-02-10T06:14:57.197 INFO:teuthology.orchestra.run.smithi173.stderr:0+1 records in 2024-02-10T06:14:57.197 INFO:teuthology.orchestra.run.smithi173.stderr:0+1 records out 2024-02-10T06:14:57.197 INFO:teuthology.orchestra.run.smithi173.stderr:70 bytes copied, 2.1329e-05 s, 3.3 MB/s 2024-02-10T06:14:57.198 INFO:teuthology.run_tasks:Running task print... 2024-02-10T06:14:57.412 INFO:teuthology.task.print:**** done client 2024-02-10T06:14:57.412 INFO:teuthology.run_tasks:Running task parallel... 2024-02-10T06:14:57.485 INFO:teuthology.task.parallel:starting parallel... 2024-02-10T06:14:57.486 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-10T06:14:57.486 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-10T06:14:57.486 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T06:14:57.487 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-02-10T06:14:57.487 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-10T06:14:57.487 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-02-10T06:14:57.493 INFO:tasks.workunit:Pulling workunits from ref 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:14:57.493 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-10T06:14:57.494 INFO:tasks.workunit:timeout=3h 2024-02-10T06:14:57.494 INFO:tasks.workunit:cleanup=True 2024-02-10T06:14:57.494 DEBUG:teuthology.orchestra.run.smithi089:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-10T06:14:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-02-10T06:14:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-02-10T06:14:57.512 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 89h/137d Inode: 1 Links: 2 2024-02-10T06:14:57.512 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:14:57.512 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-10T06:14:57.512 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-10T06:14:57.512 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-02-10 06:14:23.003204653 +0000 2024-02-10T06:14:57.512 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-02-10 06:14:52.508709714 +0000 2024-02-10T06:14:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-02-10 06:14:23.003204653 +0000 2024-02-10T06:14:57.513 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-02-10T06:14:57.513 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-02-10T06:14:57.521 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:57 smithi089 ceph-mon[29217]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-02-10T06:14:57.521 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:57 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:57.551 DEBUG:teuthology.orchestra.run.smithi173:> stat -- /home/ubuntu/cephtest/mnt.1 2024-02-10T06:14:57.569 INFO:teuthology.orchestra.run.smithi173.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-02-10T06:14:57.569 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-02-10T06:14:57.569 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-02-10T06:14:57.569 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T06:14:57.569 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-10T06:14:57.569 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-10T06:14:57.569 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-02-10 06:14:57.545594026 +0000 2024-02-10T06:14:57.570 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-02-10 06:14:57.545594026 +0000 2024-02-10T06:14:57.570 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-02-10 06:14:23.003204653 +0000 2024-02-10T06:14:57.570 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-02-10T06:14:57.570 DEBUG:teuthology.orchestra.run.smithi173:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-02-10T06:14:57.632 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:57 smithi173 ceph-mon[36251]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-02-10T06:14:57.633 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:57 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:57.639 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:14:57.640 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:14:57.657 INFO:tasks.workunit.client.0.smithi089.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-10T06:14:57.695 INFO:tasks.workunit.client.1.smithi173.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-02-10T06:14:57.770 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:14:59.104 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:14:59 smithi089 ceph-mon[29217]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-02-10T06:14:59.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.347+0000 7f6493b59700 1 -- 172.21.15.89:0/115508660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c0fe0e0 msgr2=0x7f648c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:59.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.347+0000 7f6493b59700 1 --2- 172.21.15.89:0/115508660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c0fe0e0 0x7f648c0fe550 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6488004940 tx=0x7f6488009e30 comp rx=0 tx=0).stop 2024-02-10T06:14:59.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.347+0000 7f6493b59700 1 -- 172.21.15.89:0/115508660 shutdown_connections 2024-02-10T06:14:59.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.347+0000 7f6493b59700 1 --2- 172.21.15.89:0/115508660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c0fe0e0 0x7f648c0fe550 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:59.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.347+0000 7f6493b59700 1 --2- 172.21.15.89:0/115508660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f648c0fcef0 0x7f648c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.347+0000 7f6493b59700 1 -- 172.21.15.89:0/115508660 >> 172.21.15.89:0/115508660 conn(0x7f648c0f8620 msgr2=0x7f648c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:14:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.347+0000 7f6493b59700 1 -- 172.21.15.89:0/115508660 shutdown_connections 2024-02-10T06:14:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.347+0000 7f6493b59700 1 -- 172.21.15.89:0/115508660 wait complete. 2024-02-10T06:14:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f6493b59700 1 Processor -- start 2024-02-10T06:14:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f6493b59700 1 -- start start 2024-02-10T06:14:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f6493b59700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c0fcef0 0x7f648c071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f6493b59700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f648c0fe0e0 0x7f648c0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f6493b59700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c070970 con 0x7f648c0fcef0 2024-02-10T06:14:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f6493b59700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c070ae0 con 0x7f648c0fe0e0 2024-02-10T06:14:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f64918f5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c0fcef0 0x7f648c071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f64918f5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c0fcef0 0x7f648c071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34298/0 (socket says 172.21.15.89:34298) 2024-02-10T06:14:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f64918f5700 1 -- 172.21.15.89:0/3478595074 learned_addr learned my addr 172.21.15.89:0/3478595074 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:14:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f64910f4700 1 --2- 172.21.15.89:0/3478595074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f648c0fe0e0 0x7f648c0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f64918f5700 1 -- 172.21.15.89:0/3478595074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f648c0fe0e0 msgr2=0x7f648c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:14:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f64918f5700 1 --2- 172.21.15.89:0/3478595074 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f648c0fe0e0 0x7f648c0702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:14:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.348+0000 7f64918f5700 1 -- 172.21.15.89:0/3478595074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6488005040 con 0x7f648c0fcef0 2024-02-10T06:14:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.349+0000 7f64918f5700 1 --2- 172.21.15.89:0/3478595074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f648c0fcef0 0x7f648c071e60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f647c00b790 tx=0x7f647c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.349+0000 7f64827fc700 1 -- 172.21.15.89:0/3478595074 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f647c00f6f0 con 0x7f648c0fcef0 2024-02-10T06:14:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.349+0000 7f6493b59700 1 -- 172.21.15.89:0/3478595074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648c070dc0 con 0x7f648c0fcef0 2024-02-10T06:14:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.349+0000 7f64827fc700 1 -- 172.21.15.89:0/3478595074 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f647c00fd20 con 0x7f648c0fcef0 2024-02-10T06:14:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.349+0000 7f64827fc700 1 -- 172.21.15.89:0/3478595074 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f647c0103a0 con 0x7f648c0fcef0 2024-02-10T06:14:59.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.349+0000 7f6493b59700 1 -- 172.21.15.89:0/3478595074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648c071140 con 0x7f648c0fcef0 2024-02-10T06:14:59.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.350+0000 7f6493b59700 1 -- 172.21.15.89:0/3478595074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648c04bed0 con 0x7f648c0fcef0 2024-02-10T06:14:59.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.350+0000 7f64827fc700 1 -- 172.21.15.89:0/3478595074 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f647c011400 con 0x7f648c0fcef0 2024-02-10T06:14:59.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.350+0000 7f64827fc700 1 --2- 172.21.15.89:0/3478595074 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f64780692a0 0x7f647806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:14:59.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.351+0000 7f64827fc700 1 -- 172.21.15.89:0/3478595074 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f647c087080 con 0x7f648c0fcef0 2024-02-10T06:14:59.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.351+0000 7f64910f4700 1 --2- 172.21.15.89:0/3478595074 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f64780692a0 0x7f647806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:14:59.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.351+0000 7f64910f4700 1 --2- 172.21.15.89:0/3478595074 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f64780692a0 0x7f647806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6488004f90 tx=0x7f6488009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:14:59.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.355+0000 7f64827fc700 1 -- 172.21.15.89:0/3478595074 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f647c056d70 con 0x7f648c0fcef0 2024-02-10T06:14:59.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:14:59 smithi173 ceph-mon[36251]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-02-10T06:14:59.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.538+0000 7f6493b59700 1 -- 172.21.15.89:0/3478595074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f648c105ad0 con 0x7f648c0fcef0 2024-02-10T06:14:59.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:14:59.538+0000 7f64827fc700 1 -- 172.21.15.89:0/3478595074 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21)=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21) v1 ==== 179+0+0 (secure 0 0 0) 0x7f648c105ad0 con 0x7f648c0fcef0 2024-02-10T06:14:59.540 INFO:teuthology.orchestra.run.smithi089.stderr:Error EINVAL: unrecognized config option 'mgr/orchestrator/fail_fs' 2024-02-10T06:15:00.164 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-10T06:15:00.164 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T06:15:00.164 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-10T06:15:00.459 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:15:01.537 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:01 smithi089 ceph-mon[29217]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-10T06:15:01.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:01 smithi173 ceph-mon[36251]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-10T06:15:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.142+0000 7f270459a700 1 -- 172.21.15.89:0/933747395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26fc0fe2b0 msgr2=0x7f26fc0fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.142+0000 7f270459a700 1 --2- 172.21.15.89:0/933747395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26fc0fe2b0 0x7f26fc0fe700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f26ec004940 tx=0x7f26ec009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.143+0000 7f270459a700 1 -- 172.21.15.89:0/933747395 shutdown_connections 2024-02-10T06:15:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.143+0000 7f270459a700 1 --2- 172.21.15.89:0/933747395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26fc0fe2b0 0x7f26fc0fe700 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.143+0000 7f270459a700 1 --2- 172.21.15.89:0/933747395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26fc0fb220 0x7f26fc0fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.143+0000 7f270459a700 1 -- 172.21.15.89:0/933747395 >> 172.21.15.89:0/933747395 conn(0x7f26fc0f6ca0 msgr2=0x7f26fc0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.143+0000 7f270459a700 1 -- 172.21.15.89:0/933747395 shutdown_connections 2024-02-10T06:15:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.143+0000 7f270459a700 1 -- 172.21.15.89:0/933747395 wait complete. 2024-02-10T06:15:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.143+0000 7f270459a700 1 Processor -- start 2024-02-10T06:15:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.143+0000 7f270459a700 1 -- start start 2024-02-10T06:15:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f270459a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26fc0fb220 0x7f26fc10d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f270459a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26fc0fe2b0 0x7f26fc10de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f270459a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc10e490 con 0x7f26fc0fb220 2024-02-10T06:15:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f270459a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc10ed30 con 0x7f26fc0fe2b0 2024-02-10T06:15:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f2702336700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26fc0fb220 0x7f26fc10d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f2702336700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26fc0fb220 0x7f26fc10d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34322/0 (socket says 172.21.15.89:34322) 2024-02-10T06:15:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f2702336700 1 -- 172.21.15.89:0/3584737445 learned_addr learned my addr 172.21.15.89:0/3584737445 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f2701b35700 1 --2- 172.21.15.89:0/3584737445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26fc0fe2b0 0x7f26fc10de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f2702336700 1 -- 172.21.15.89:0/3584737445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26fc0fe2b0 msgr2=0x7f26fc10de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f2702336700 1 --2- 172.21.15.89:0/3584737445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26fc0fe2b0 0x7f26fc10de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:02.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f2702336700 1 -- 172.21.15.89:0/3584737445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec005040 con 0x7f26fc0fb220 2024-02-10T06:15:02.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.144+0000 7f2702336700 1 --2- 172.21.15.89:0/3584737445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26fc0fb220 0x7f26fc10d950 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f26f800eb70 tx=0x7f26f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:02.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.145+0000 7f26f2ffd700 1 -- 172.21.15.89:0/3584737445 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26f8007cc0 con 0x7f26fc0fb220 2024-02-10T06:15:02.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.145+0000 7f26f2ffd700 1 -- 172.21.15.89:0/3584737445 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26f800f040 con 0x7f26fc0fb220 2024-02-10T06:15:02.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.145+0000 7f270459a700 1 -- 172.21.15.89:0/3584737445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc10efb0 con 0x7f26fc0fb220 2024-02-10T06:15:02.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.145+0000 7f26f2ffd700 1 -- 172.21.15.89:0/3584737445 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26f800cb20 con 0x7f26fc0fb220 2024-02-10T06:15:02.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.145+0000 7f270459a700 1 -- 172.21.15.89:0/3584737445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc10f4f0 con 0x7f26fc0fb220 2024-02-10T06:15:02.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.146+0000 7f270459a700 1 -- 172.21.15.89:0/3584737445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc04bed0 con 0x7f26fc0fb220 2024-02-10T06:15:02.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.146+0000 7f26f2ffd700 1 -- 172.21.15.89:0/3584737445 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f26f800cc80 con 0x7f26fc0fb220 2024-02-10T06:15:02.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.147+0000 7f26f2ffd700 1 --2- 172.21.15.89:0/3584737445 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f26e80692a0 0x7f26e806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:02.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.147+0000 7f26f2ffd700 1 -- 172.21.15.89:0/3584737445 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f26f8086b20 con 0x7f26fc0fb220 2024-02-10T06:15:02.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.147+0000 7f2701b35700 1 --2- 172.21.15.89:0/3584737445 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f26e80692a0 0x7f26e806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:02.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.147+0000 7f2701b35700 1 --2- 172.21.15.89:0/3584737445 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f26e80692a0 0x7f26e806b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00b0d0 tx=0x7f26ec009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:02.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.152+0000 7f26f2ffd700 1 -- 172.21.15.89:0/3584737445 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f26f804dbd0 con 0x7f26fc0fb220 2024-02-10T06:15:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.347+0000 7f270459a700 1 -- 172.21.15.89:0/3584737445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f26fc105a40 con 0x7f26fc0fb220 2024-02-10T06:15:02.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.347+0000 7f26f2ffd700 1 -- 172.21.15.89:0/3584737445 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f26f8056940 con 0x7f26fc0fb220 2024-02-10T06:15:02.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.350+0000 7f270459a700 1 -- 172.21.15.89:0/3584737445 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f26e80692a0 msgr2=0x7f26e806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:02.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.350+0000 7f270459a700 1 --2- 172.21.15.89:0/3584737445 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f26e80692a0 0x7f26e806b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00b0d0 tx=0x7f26ec009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:02.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.350+0000 7f270459a700 1 -- 172.21.15.89:0/3584737445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26fc0fb220 msgr2=0x7f26fc10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:02.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.350+0000 7f270459a700 1 --2- 172.21.15.89:0/3584737445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26fc0fb220 0x7f26fc10d950 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f26f800eb70 tx=0x7f26f800b040 comp rx=0 tx=0).stop 2024-02-10T06:15:02.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.350+0000 7f270459a700 1 -- 172.21.15.89:0/3584737445 shutdown_connections 2024-02-10T06:15:02.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.350+0000 7f270459a700 1 --2- 172.21.15.89:0/3584737445 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f26e80692a0 0x7f26e806b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:02.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.350+0000 7f270459a700 1 --2- 172.21.15.89:0/3584737445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26fc0fb220 0x7f26fc10d950 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:02.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.350+0000 7f270459a700 1 --2- 172.21.15.89:0/3584737445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26fc0fe2b0 0x7f26fc10de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:02.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.350+0000 7f270459a700 1 -- 172.21.15.89:0/3584737445 >> 172.21.15.89:0/3584737445 conn(0x7f26fc0f6ca0 msgr2=0x7f26fc1014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:02.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.350+0000 7f270459a700 1 -- 172.21.15.89:0/3584737445 shutdown_connections 2024-02-10T06:15:02.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:02.350+0000 7f270459a700 1 -- 172.21.15.89:0/3584737445 wait complete. 2024-02-10T06:15:03.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-10T06:15:03.344 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:15:03.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:03 smithi089 ceph-mon[29217]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:15:03.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:03 smithi173 ceph-mon[36251]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:15:05.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.055+0000 7fcabef7c700 1 -- 172.21.15.89:0/3470335864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcab81000c0 msgr2=0x7fcab8102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:05.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.055+0000 7fcabef7c700 1 --2- 172.21.15.89:0/3470335864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcab81000c0 0x7fcab8102520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4004940 tx=0x7fcaa4009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:05.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.055+0000 7fcabef7c700 1 -- 172.21.15.89:0/3470335864 shutdown_connections 2024-02-10T06:15:05.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.055+0000 7fcabef7c700 1 --2- 172.21.15.89:0/3470335864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcab81000c0 0x7fcab8102520 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:05.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.055+0000 7fcabef7c700 1 --2- 172.21.15.89:0/3470335864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcab80663e0 0x7fcab80ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:05.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.055+0000 7fcabef7c700 1 -- 172.21.15.89:0/3470335864 >> 172.21.15.89:0/3470335864 conn(0x7fcab80f74e0 msgr2=0x7fcab80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:05.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.055+0000 7fcabef7c700 1 -- 172.21.15.89:0/3470335864 shutdown_connections 2024-02-10T06:15:05.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.055+0000 7fcabef7c700 1 -- 172.21.15.89:0/3470335864 wait complete. 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.056+0000 7fcabef7c700 1 Processor -- start 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.056+0000 7fcabef7c700 1 -- start start 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.056+0000 7fcabef7c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcab80663e0 0x7fcab810d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.056+0000 7fcabef7c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcab81000c0 0x7fcab810de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.056+0000 7fcabef7c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab810e410 con 0x7fcab81000c0 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.056+0000 7fcabef7c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab810ecb0 con 0x7fcab80663e0 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.056+0000 7fcab7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcab81000c0 0x7fcab810de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.056+0000 7fcab7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcab81000c0 0x7fcab810de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34340/0 (socket says 172.21.15.89:34340) 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.056+0000 7fcab7fff700 1 -- 172.21.15.89:0/710904003 learned_addr learned my addr 172.21.15.89:0/710904003 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.056+0000 7fcabcd18700 1 --2- 172.21.15.89:0/710904003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcab80663e0 0x7fcab810d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.057+0000 7fcab7fff700 1 -- 172.21.15.89:0/710904003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcab80663e0 msgr2=0x7fcab810d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:05.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.057+0000 7fcab7fff700 1 --2- 172.21.15.89:0/710904003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcab80663e0 0x7fcab810d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:05.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.057+0000 7fcab7fff700 1 -- 172.21.15.89:0/710904003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa4005040 con 0x7fcab81000c0 2024-02-10T06:15:05.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.057+0000 7fcab7fff700 1 --2- 172.21.15.89:0/710904003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcab81000c0 0x7fcab810de10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4000c00 tx=0x7fcaa4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:05.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.057+0000 7fcab57fa700 1 -- 172.21.15.89:0/710904003 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaa400e040 con 0x7fcab81000c0 2024-02-10T06:15:05.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.057+0000 7fcab57fa700 1 -- 172.21.15.89:0/710904003 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcaa400bbf0 con 0x7fcab81000c0 2024-02-10T06:15:05.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.057+0000 7fcab57fa700 1 -- 172.21.15.89:0/710904003 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaa401c630 con 0x7fcab81000c0 2024-02-10T06:15:05.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.057+0000 7fcabef7c700 1 -- 172.21.15.89:0/710904003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab810ef30 con 0x7fcab81000c0 2024-02-10T06:15:05.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.057+0000 7fcabef7c700 1 -- 172.21.15.89:0/710904003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab810f470 con 0x7fcab81000c0 2024-02-10T06:15:05.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.058+0000 7fcabef7c700 1 -- 172.21.15.89:0/710904003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab804bed0 con 0x7fcab81000c0 2024-02-10T06:15:05.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.059+0000 7fcab57fa700 1 -- 172.21.15.89:0/710904003 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcaa400bd60 con 0x7fcab81000c0 2024-02-10T06:15:05.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.059+0000 7fcab57fa700 1 --2- 172.21.15.89:0/710904003 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fcaa80692b0 0x7fcaa806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:05.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.060+0000 7fcabcd18700 1 --2- 172.21.15.89:0/710904003 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fcaa80692b0 0x7fcaa806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:05.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.060+0000 7fcab57fa700 1 -- 172.21.15.89:0/710904003 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fcaa40883b0 con 0x7fcab81000c0 2024-02-10T06:15:05.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.060+0000 7fcabcd18700 1 --2- 172.21.15.89:0/710904003 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fcaa80692b0 0x7fcaa806b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcaac009fd0 tx=0x7fcaac00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:05.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.064+0000 7fcab57fa700 1 -- 172.21.15.89:0/710904003 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcaa4058270 con 0x7fcab81000c0 2024-02-10T06:15:05.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.259+0000 7fcabef7c700 1 -- 172.21.15.89:0/710904003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fcab8105a10 con 0x7fcab81000c0 2024-02-10T06:15:05.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.259+0000 7fcab57fa700 1 -- 172.21.15.89:0/710904003 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7fcaa4013e60 con 0x7fcab81000c0 2024-02-10T06:15:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.262+0000 7fcabef7c700 1 -- 172.21.15.89:0/710904003 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fcaa80692b0 msgr2=0x7fcaa806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.262+0000 7fcabef7c700 1 --2- 172.21.15.89:0/710904003 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fcaa80692b0 0x7fcaa806b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcaac009fd0 tx=0x7fcaac00b040 comp rx=0 tx=0).stop 2024-02-10T06:15:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.262+0000 7fcabef7c700 1 -- 172.21.15.89:0/710904003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcab81000c0 msgr2=0x7fcab810de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.262+0000 7fcabef7c700 1 --2- 172.21.15.89:0/710904003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcab81000c0 0x7fcab810de10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4000c00 tx=0x7fcaa4009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.262+0000 7fcabef7c700 1 -- 172.21.15.89:0/710904003 shutdown_connections 2024-02-10T06:15:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.262+0000 7fcabef7c700 1 --2- 172.21.15.89:0/710904003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcab80663e0 0x7fcab810d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.262+0000 7fcabef7c700 1 --2- 172.21.15.89:0/710904003 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fcaa80692b0 0x7fcaa806b760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.262+0000 7fcabef7c700 1 --2- 172.21.15.89:0/710904003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcab81000c0 0x7fcab810de10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.262+0000 7fcabef7c700 1 -- 172.21.15.89:0/710904003 >> 172.21.15.89:0/710904003 conn(0x7fcab80f74e0 msgr2=0x7fcab80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:05.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.262+0000 7fcabef7c700 1 -- 172.21.15.89:0/710904003 shutdown_connections 2024-02-10T06:15:05.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:05.262+0000 7fcabef7c700 1 -- 172.21.15.89:0/710904003 wait complete. 2024-02-10T06:15:05.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:05 smithi089 ceph-mon[29217]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-10T06:15:05.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:05 smithi173 ceph-mon[36251]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-10T06:15:06.004 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-10T06:15:06.281 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:15:07.561 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:07 smithi089 ceph-mon[29217]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-10T06:15:07.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:07 smithi173 ceph-mon[36251]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-10T06:15:07.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.994+0000 7f4de81b7700 1 -- 172.21.15.89:0/377398568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4de00fcfc0 msgr2=0x7f4de00fd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.994+0000 7f4de81b7700 1 --2- 172.21.15.89:0/377398568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4de00fcfc0 0x7f4de00fd430 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0004940 tx=0x7f4dd0009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.994+0000 7f4de81b7700 1 -- 172.21.15.89:0/377398568 shutdown_connections 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.994+0000 7f4de81b7700 1 --2- 172.21.15.89:0/377398568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4de00fcfc0 0x7f4de00fd430 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.994+0000 7f4de81b7700 1 --2- 172.21.15.89:0/377398568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de00fbe50 0x7f4de00fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.994+0000 7f4de81b7700 1 -- 172.21.15.89:0/377398568 >> 172.21.15.89:0/377398568 conn(0x7f4de00f7510 msgr2=0x7f4de00f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.994+0000 7f4de81b7700 1 -- 172.21.15.89:0/377398568 shutdown_connections 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.994+0000 7f4de81b7700 1 -- 172.21.15.89:0/377398568 wait complete. 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.995+0000 7f4de81b7700 1 Processor -- start 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.995+0000 7f4de81b7700 1 -- start start 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.995+0000 7f4de81b7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de00fbe50 0x7f4de01105d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.995+0000 7f4de81b7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4de00fcfc0 0x7f4de0112b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:07.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.995+0000 7f4de81b7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de0066cf0 con 0x7f4de00fcfc0 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.995+0000 7f4de81b7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de0066e30 con 0x7f4de00fbe50 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.995+0000 7f4de5752700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4de00fcfc0 0x7f4de0112b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.995+0000 7f4de5752700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4de00fcfc0 0x7f4de0112b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34350/0 (socket says 172.21.15.89:34350) 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.995+0000 7f4de5f53700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de00fbe50 0x7f4de01105d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.995+0000 7f4de5752700 1 -- 172.21.15.89:0/1699856888 learned_addr learned my addr 172.21.15.89:0/1699856888 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.996+0000 7f4de5752700 1 -- 172.21.15.89:0/1699856888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de00fbe50 msgr2=0x7f4de01105d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.996+0000 7f4de5752700 1 --2- 172.21.15.89:0/1699856888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de00fbe50 0x7f4de01105d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.996+0000 7f4de5752700 1 -- 172.21.15.89:0/1699856888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd0005040 con 0x7f4de00fcfc0 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.996+0000 7f4de5752700 1 --2- 172.21.15.89:0/1699856888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4de00fcfc0 0x7f4de0112b20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0000c00 tx=0x7f4dd0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.996+0000 7f4dd67fc700 1 -- 172.21.15.89:0/1699856888 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dd0021070 con 0x7f4de00fcfc0 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.996+0000 7f4dd67fc700 1 -- 172.21.15.89:0/1699856888 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4dd001c400 con 0x7f4de00fcfc0 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.996+0000 7f4de81b7700 1 -- 172.21.15.89:0/1699856888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de01130c0 con 0x7f4de00fcfc0 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.996+0000 7f4dd67fc700 1 -- 172.21.15.89:0/1699856888 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dd000e070 con 0x7f4de00fcfc0 2024-02-10T06:15:07.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.996+0000 7f4de81b7700 1 -- 172.21.15.89:0/1699856888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de0113600 con 0x7f4de00fcfc0 2024-02-10T06:15:07.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.997+0000 7f4de81b7700 1 -- 172.21.15.89:0/1699856888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de004bed0 con 0x7f4de00fcfc0 2024-02-10T06:15:07.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.997+0000 7f4dd67fc700 1 -- 172.21.15.89:0/1699856888 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4dd001ca50 con 0x7f4de00fcfc0 2024-02-10T06:15:08.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.998+0000 7f4dd67fc700 1 --2- 172.21.15.89:0/1699856888 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4dcc0692b0 0x7f4dcc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:08.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.998+0000 7f4dd67fc700 1 -- 172.21.15.89:0/1699856888 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4dd0088300 con 0x7f4de00fcfc0 2024-02-10T06:15:08.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.998+0000 7f4de5f53700 1 --2- 172.21.15.89:0/1699856888 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4dcc0692b0 0x7f4dcc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:08.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:07.998+0000 7f4de5f53700 1 --2- 172.21.15.89:0/1699856888 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4dcc0692b0 0x7f4dcc06b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4de0066970 tx=0x7f4ddc00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:08.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.003+0000 7f4dd67fc700 1 -- 172.21.15.89:0/1699856888 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4dd0054680 con 0x7f4de00fcfc0 2024-02-10T06:15:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.210+0000 7f4de81b7700 1 -- 172.21.15.89:0/1699856888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f4de0104940 con 0x7f4de00fcfc0 2024-02-10T06:15:08.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.210+0000 7f4dd67fc700 1 -- 172.21.15.89:0/1699856888 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f4dd0057ca0 con 0x7f4de00fcfc0 2024-02-10T06:15:08.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.213+0000 7f4de81b7700 1 -- 172.21.15.89:0/1699856888 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4dcc0692b0 msgr2=0x7f4dcc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:08.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.213+0000 7f4de81b7700 1 --2- 172.21.15.89:0/1699856888 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4dcc0692b0 0x7f4dcc06b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4de0066970 tx=0x7f4ddc00b040 comp rx=0 tx=0).stop 2024-02-10T06:15:08.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.213+0000 7f4de81b7700 1 -- 172.21.15.89:0/1699856888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4de00fcfc0 msgr2=0x7f4de0112b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:08.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.213+0000 7f4de81b7700 1 --2- 172.21.15.89:0/1699856888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4de00fcfc0 0x7f4de0112b20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0000c00 tx=0x7f4dd0009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:08.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.213+0000 7f4de81b7700 1 -- 172.21.15.89:0/1699856888 shutdown_connections 2024-02-10T06:15:08.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.213+0000 7f4de81b7700 1 --2- 172.21.15.89:0/1699856888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4de00fbe50 0x7f4de01105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:08.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.213+0000 7f4de81b7700 1 --2- 172.21.15.89:0/1699856888 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f4dcc0692b0 0x7f4dcc06b760 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:08.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.213+0000 7f4de81b7700 1 --2- 172.21.15.89:0/1699856888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4de00fcfc0 0x7f4de0112b20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:08.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.213+0000 7f4de81b7700 1 -- 172.21.15.89:0/1699856888 >> 172.21.15.89:0/1699856888 conn(0x7f4de00f7510 msgr2=0x7f4de0100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:08.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.213+0000 7f4de81b7700 1 -- 172.21.15.89:0/1699856888 shutdown_connections 2024-02-10T06:15:08.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:08.213+0000 7f4de81b7700 1 -- 172.21.15.89:0/1699856888 wait complete. 2024-02-10T06:15:08.913 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-10T06:15:09.220 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:15:09.346 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:09 smithi089 ceph-mon[29217]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-10T06:15:09.399 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:09 smithi173 ceph-mon[36251]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-10T06:15:10.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.917+0000 7f68c4568700 1 -- 172.21.15.89:0/3586667468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc100120 msgr2=0x7f68bc102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:10.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.917+0000 7f68c4568700 1 --2- 172.21.15.89:0/3586667468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc100120 0x7f68bc102520 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f68ac004940 tx=0x7f68ac009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:10.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.917+0000 7f68c4568700 1 -- 172.21.15.89:0/3586667468 shutdown_connections 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.917+0000 7f68c4568700 1 --2- 172.21.15.89:0/3586667468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc100120 0x7f68bc102520 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.917+0000 7f68c4568700 1 --2- 172.21.15.89:0/3586667468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68bc066c10 0x7f68bc0ffb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.917+0000 7f68c4568700 1 -- 172.21.15.89:0/3586667468 >> 172.21.15.89:0/3586667468 conn(0x7f68bc0f7500 msgr2=0x7f68bc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.917+0000 7f68c4568700 1 -- 172.21.15.89:0/3586667468 shutdown_connections 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.917+0000 7f68c4568700 1 -- 172.21.15.89:0/3586667468 wait complete. 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.917+0000 7f68c4568700 1 Processor -- start 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.917+0000 7f68c4568700 1 -- start start 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c4568700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68bc066c10 0x7f68bc10d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c4568700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc100120 0x7f68bc10de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c4568700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc10e480 con 0x7f68bc100120 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c4568700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc10ed20 con 0x7f68bc066c10 2024-02-10T06:15:10.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c1b03700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc100120 0x7f68bc10de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c1b03700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc100120 0x7f68bc10de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37198/0 (socket says 172.21.15.89:37198) 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c1b03700 1 -- 172.21.15.89:0/2534722105 learned_addr learned my addr 172.21.15.89:0/2534722105 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c2304700 1 --2- 172.21.15.89:0/2534722105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68bc066c10 0x7f68bc10d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c1b03700 1 -- 172.21.15.89:0/2534722105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68bc066c10 msgr2=0x7f68bc10d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c1b03700 1 --2- 172.21.15.89:0/2534722105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68bc066c10 0x7f68bc10d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c1b03700 1 -- 172.21.15.89:0/2534722105 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac005040 con 0x7f68bc100120 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68c1b03700 1 --2- 172.21.15.89:0/2534722105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc100120 0x7f68bc10de80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f68ac000c00 tx=0x7f68ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.918+0000 7f68b2ffd700 1 -- 172.21.15.89:0/2534722105 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68ac00e040 con 0x7f68bc100120 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.919+0000 7f68c4568700 1 -- 172.21.15.89:0/2534722105 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68bc10efa0 con 0x7f68bc100120 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.919+0000 7f68b2ffd700 1 -- 172.21.15.89:0/2534722105 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68ac00bbf0 con 0x7f68bc100120 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.919+0000 7f68b2ffd700 1 -- 172.21.15.89:0/2534722105 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68ac01c630 con 0x7f68bc100120 2024-02-10T06:15:10.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.919+0000 7f68c4568700 1 -- 172.21.15.89:0/2534722105 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68bc10f4e0 con 0x7f68bc100120 2024-02-10T06:15:10.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.920+0000 7f68c4568700 1 -- 172.21.15.89:0/2534722105 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68bc04bed0 con 0x7f68bc100120 2024-02-10T06:15:10.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.920+0000 7f68b2ffd700 1 -- 172.21.15.89:0/2534722105 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f68ac00bd60 con 0x7f68bc100120 2024-02-10T06:15:10.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.921+0000 7f68b2ffd700 1 --2- 172.21.15.89:0/2534722105 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f68a80692b0 0x7f68a806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:10.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.921+0000 7f68c2304700 1 --2- 172.21.15.89:0/2534722105 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f68a80692b0 0x7f68a806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:10.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.921+0000 7f68b2ffd700 1 -- 172.21.15.89:0/2534722105 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f68ac088470 con 0x7f68bc100120 2024-02-10T06:15:10.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.921+0000 7f68c2304700 1 --2- 172.21.15.89:0/2534722105 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f68a80692b0 0x7f68a806b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f68b8004f40 tx=0x7f68b800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:10.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:10.925+0000 7f68b2ffd700 1 -- 172.21.15.89:0/2534722105 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f68ac058290 con 0x7f68bc100120 2024-02-10T06:15:11.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.169+0000 7f68c4568700 1 -- 172.21.15.89:0/2534722105 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}) v1 -- 0x7f68bc10f990 con 0x7f68a80692b0 2024-02-10T06:15:11.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.174+0000 7f68b2ffd700 1 -- 172.21.15.89:0/2534722105 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f68bc10f990 con 0x7f68a80692b0 2024-02-10T06:15:11.175 INFO:teuthology.orchestra.run.smithi089.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:15:11.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.176+0000 7f68c4568700 1 -- 172.21.15.89:0/2534722105 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f68a80692b0 msgr2=0x7f68a806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:11.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.176+0000 7f68c4568700 1 --2- 172.21.15.89:0/2534722105 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f68a80692b0 0x7f68a806b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f68b8004f40 tx=0x7f68b800b040 comp rx=0 tx=0).stop 2024-02-10T06:15:11.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.176+0000 7f68c4568700 1 -- 172.21.15.89:0/2534722105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc100120 msgr2=0x7f68bc10de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:11.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.176+0000 7f68c4568700 1 --2- 172.21.15.89:0/2534722105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc100120 0x7f68bc10de80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f68ac000c00 tx=0x7f68ac009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:11.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.176+0000 7f68c4568700 1 -- 172.21.15.89:0/2534722105 shutdown_connections 2024-02-10T06:15:11.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.176+0000 7f68c4568700 1 --2- 172.21.15.89:0/2534722105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68bc066c10 0x7f68bc10d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:11.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.176+0000 7f68c4568700 1 --2- 172.21.15.89:0/2534722105 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f68a80692b0 0x7f68a806b760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:11.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.176+0000 7f68c4568700 1 --2- 172.21.15.89:0/2534722105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68bc100120 0x7f68bc10de80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:11.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.176+0000 7f68c4568700 1 -- 172.21.15.89:0/2534722105 >> 172.21.15.89:0/2534722105 conn(0x7f68bc0f7500 msgr2=0x7f68bc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:11.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.176+0000 7f68c4568700 1 -- 172.21.15.89:0/2534722105 shutdown_connections 2024-02-10T06:15:11.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:11.176+0000 7f68c4568700 1 -- 172.21.15.89:0/2534722105 wait complete. 2024-02-10T06:15:11.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:11 smithi089 ceph-mon[29217]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-10T06:15:11.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:11 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:15:11.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:11 smithi173 ceph-mon[36251]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-10T06:15:11.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:11 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:15:11.891 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-10T06:15:11.892 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T06:15:11.892 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-10T06:15:12.202 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:15:12.487 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:12 smithi089 ceph-mon[29217]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:12.487 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:12 smithi089 ceph-mon[29217]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:15:12.487 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:12.487 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:12.487 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:12.487 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:15:12.487 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:12 smithi089 ceph-mon[29217]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:15:12.487 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:12 smithi173 ceph-mon[36251]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:12 smithi173 ceph-mon[36251]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:15:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:15:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:12 smithi173 ceph-mon[36251]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T06:15:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:13.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:13 smithi089 ceph-mon[29217]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:13.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:13 smithi173 ceph-mon[36251]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:13.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.953+0000 7fdf08f80700 1 -- 172.21.15.89:0/201645529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf040fcfb0 msgr2=0x7fdf040fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:13.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.953+0000 7fdf08f80700 1 --2- 172.21.15.89:0/201645529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf040fcfb0 0x7fdf040fd420 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdef8004940 tx=0x7fdef8009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:13.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.953+0000 7fdf08f80700 1 -- 172.21.15.89:0/201645529 shutdown_connections 2024-02-10T06:15:13.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.953+0000 7fdf08f80700 1 --2- 172.21.15.89:0/201645529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf040fcfb0 0x7fdf040fd420 secure :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdef8004940 tx=0x7fdef8009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:13.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.953+0000 7fdf08f80700 1 --2- 172.21.15.89:0/201645529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf040fbe40 0x7fdf040fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:13.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.953+0000 7fdf08f80700 1 -- 172.21.15.89:0/201645529 >> 172.21.15.89:0/201645529 conn(0x7fdf040f74e0 msgr2=0x7fdf040f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:13.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.953+0000 7fdf08f80700 1 -- 172.21.15.89:0/201645529 shutdown_connections 2024-02-10T06:15:13.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.953+0000 7fdf08f80700 1 -- 172.21.15.89:0/201645529 wait complete. 2024-02-10T06:15:13.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.953+0000 7fdf08f80700 1 Processor -- start 2024-02-10T06:15:13.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.954+0000 7fdf08f80700 1 -- start start 2024-02-10T06:15:13.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.954+0000 7fdf08f80700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf040fbe40 0x7fdf0410da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:13.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.954+0000 7fdf08f80700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf0410dfa0 0x7fdf0410ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:13.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.954+0000 7fdf08f80700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0410e460 con 0x7fdf040fbe40 2024-02-10T06:15:13.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.954+0000 7fdf08f80700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0410e5a0 con 0x7fdf0410dfa0 2024-02-10T06:15:13.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.954+0000 7fdf0259c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf040fbe40 0x7fdf0410da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:13.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.954+0000 7fdf01d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf0410dfa0 0x7fdf0410ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:13.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.954+0000 7fdf01d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf0410dfa0 0x7fdf0410ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.89:43256/0 (socket says 172.21.15.89:43256) 2024-02-10T06:15:13.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.954+0000 7fdf01d9b700 1 -- 172.21.15.89:0/758447620 learned_addr learned my addr 172.21.15.89:0/758447620 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:13.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.955+0000 7fdf01d9b700 1 -- 172.21.15.89:0/758447620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf040fbe40 msgr2=0x7fdf0410da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:13.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.955+0000 7fdf01d9b700 1 --2- 172.21.15.89:0/758447620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf040fbe40 0x7fdf0410da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:13.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.955+0000 7fdf01d9b700 1 -- 172.21.15.89:0/758447620 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef8005040 con 0x7fdf0410dfa0 2024-02-10T06:15:13.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.955+0000 7fdf0259c700 1 --2- 172.21.15.89:0/758447620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf040fbe40 0x7fdf0410da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:15:13.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.955+0000 7fdf01d9b700 1 --2- 172.21.15.89:0/758447620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf0410dfa0 0x7fdf0410ee40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdef8000c00 tx=0x7fdef8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:13.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.955+0000 7fdef2ffd700 1 -- 172.21.15.89:0/758447620 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdef8021070 con 0x7fdf0410dfa0 2024-02-10T06:15:13.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.955+0000 7fdef2ffd700 1 -- 172.21.15.89:0/758447620 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdef8013de0 con 0x7fdf0410dfa0 2024-02-10T06:15:13.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.955+0000 7fdf08f80700 1 -- 172.21.15.89:0/758447620 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0410f440 con 0x7fdf0410dfa0 2024-02-10T06:15:13.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.956+0000 7fdef2ffd700 1 -- 172.21.15.89:0/758447620 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdef800e070 con 0x7fdf0410dfa0 2024-02-10T06:15:13.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.956+0000 7fdf08f80700 1 -- 172.21.15.89:0/758447620 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0410f890 con 0x7fdf0410dfa0 2024-02-10T06:15:13.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.958+0000 7fdef2ffd700 1 -- 172.21.15.89:0/758447620 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fdef801c690 con 0x7fdf0410dfa0 2024-02-10T06:15:13.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.958+0000 7fdef2ffd700 1 --2- 172.21.15.89:0/758447620 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdeec069380 0x7fdeec06b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:13.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.958+0000 7fdef2ffd700 1 -- 172.21.15.89:0/758447620 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdef8088320 con 0x7fdf0410dfa0 2024-02-10T06:15:13.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.958+0000 7fdf0259c700 1 --2- 172.21.15.89:0/758447620 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdeec069380 0x7fdeec06b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:13.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.959+0000 7fdf0259c700 1 --2- 172.21.15.89:0/758447620 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdeec069380 0x7fdeec06b830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009b80 tx=0x7fdef400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:13.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.959+0000 7fdf08f80700 1 -- 172.21.15.89:0/758447620 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0404bed0 con 0x7fdf0410dfa0 2024-02-10T06:15:13.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:13.964+0000 7fdef2ffd700 1 -- 172.21.15.89:0/758447620 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdef8058070 con 0x7fdf0410dfa0 2024-02-10T06:15:14.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.304+0000 7fdf08f80700 1 -- 172.21.15.89:0/758447620 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf04065dd0 con 0x7fdeec069380 2024-02-10T06:15:14.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.305+0000 7fdef2ffd700 1 -- 172.21.15.89:0/758447620 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fdf04065dd0 con 0x7fdeec069380 2024-02-10T06:15:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.308+0000 7fdf08f80700 1 -- 172.21.15.89:0/758447620 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdeec069380 msgr2=0x7fdeec06b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.308+0000 7fdf08f80700 1 --2- 172.21.15.89:0/758447620 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdeec069380 0x7fdeec06b830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009b80 tx=0x7fdef400b040 comp rx=0 tx=0).stop 2024-02-10T06:15:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.308+0000 7fdf08f80700 1 -- 172.21.15.89:0/758447620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf0410dfa0 msgr2=0x7fdf0410ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.308+0000 7fdf08f80700 1 --2- 172.21.15.89:0/758447620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf0410dfa0 0x7fdf0410ee40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdef8000c00 tx=0x7fdef8009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.308+0000 7fdf08f80700 1 -- 172.21.15.89:0/758447620 shutdown_connections 2024-02-10T06:15:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.308+0000 7fdf08f80700 1 --2- 172.21.15.89:0/758447620 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdeec069380 0x7fdeec06b830 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.308+0000 7fdf08f80700 1 --2- 172.21.15.89:0/758447620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf040fbe40 0x7fdf0410da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.308+0000 7fdf08f80700 1 --2- 172.21.15.89:0/758447620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf0410dfa0 0x7fdf0410ee40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.308+0000 7fdf08f80700 1 -- 172.21.15.89:0/758447620 >> 172.21.15.89:0/758447620 conn(0x7fdf040f74e0 msgr2=0x7fdf040656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.308+0000 7fdf08f80700 1 -- 172.21.15.89:0/758447620 shutdown_connections 2024-02-10T06:15:14.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.308+0000 7fdf08f80700 1 -- 172.21.15.89:0/758447620 wait complete. 2024-02-10T06:15:14.323 INFO:teuthology.orchestra.run.smithi089.stdout:true 2024-02-10T06:15:14.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.417+0000 7f0780f3f700 1 -- 172.21.15.89:0/4035410322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0fbcc0 msgr2=0x7f077c102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:14.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.417+0000 7f0780f3f700 1 --2- 172.21.15.89:0/4035410322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0fbcc0 0x7f077c102560 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0770004940 tx=0x7f0770009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:14.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.417+0000 7f0780f3f700 1 -- 172.21.15.89:0/4035410322 shutdown_connections 2024-02-10T06:15:14.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.417+0000 7f0780f3f700 1 --2- 172.21.15.89:0/4035410322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0fbcc0 0x7f077c102560 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.417+0000 7f0780f3f700 1 --2- 172.21.15.89:0/4035410322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f077c0fb2e0 0x7f077c0fb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.417+0000 7f0780f3f700 1 -- 172.21.15.89:0/4035410322 >> 172.21.15.89:0/4035410322 conn(0x7f077c0f6ca0 msgr2=0x7f077c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:14.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.417+0000 7f0780f3f700 1 -- 172.21.15.89:0/4035410322 shutdown_connections 2024-02-10T06:15:14.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.417+0000 7f0780f3f700 1 -- 172.21.15.89:0/4035410322 wait complete. 2024-02-10T06:15:14.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f0780f3f700 1 Processor -- start 2024-02-10T06:15:14.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f0780f3f700 1 -- start start 2024-02-10T06:15:14.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f0780f3f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0fb2e0 0x7f077c0fde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:14.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f0780f3f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f077c0fc2c0 0x7f077c0fc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:14.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f0780f3f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c0fccc0 con 0x7f077c0fb2e0 2024-02-10T06:15:14.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f0780f3f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c0fce00 con 0x7f077c0fc2c0 2024-02-10T06:15:14.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f077a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0fb2e0 0x7f077c0fde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:14.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f077a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0fb2e0 0x7f077c0fde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37228/0 (socket says 172.21.15.89:37228) 2024-02-10T06:15:14.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f077a59c700 1 -- 172.21.15.89:0/3174389991 learned_addr learned my addr 172.21.15.89:0/3174389991 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:14.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f0779d9b700 1 --2- 172.21.15.89:0/3174389991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f077c0fc2c0 0x7f077c0fc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:14.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f077a59c700 1 -- 172.21.15.89:0/3174389991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f077c0fc2c0 msgr2=0x7f077c0fc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:14.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f077a59c700 1 --2- 172.21.15.89:0/3174389991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f077c0fc2c0 0x7f077c0fc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.418+0000 7f077a59c700 1 -- 172.21.15.89:0/3174389991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0770005040 con 0x7f077c0fb2e0 2024-02-10T06:15:14.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.419+0000 7f077a59c700 1 --2- 172.21.15.89:0/3174389991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0fb2e0 0x7f077c0fde40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f076c00eb40 tx=0x7f076c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:14.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.419+0000 7f076affd700 1 -- 172.21.15.89:0/3174389991 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f076c007cc0 con 0x7f077c0fb2e0 2024-02-10T06:15:14.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.419+0000 7f076affd700 1 -- 172.21.15.89:0/3174389991 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f076c00f040 con 0x7f077c0fb2e0 2024-02-10T06:15:14.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.419+0000 7f076affd700 1 -- 172.21.15.89:0/3174389991 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f076c00ca10 con 0x7f077c0fb2e0 2024-02-10T06:15:14.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.419+0000 7f0780f3f700 1 -- 172.21.15.89:0/3174389991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077c0fd080 con 0x7f077c0fb2e0 2024-02-10T06:15:14.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.420+0000 7f0780f3f700 1 -- 172.21.15.89:0/3174389991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077c115cb0 con 0x7f077c0fb2e0 2024-02-10T06:15:14.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.421+0000 7f076affd700 1 -- 172.21.15.89:0/3174389991 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f076c00cc80 con 0x7f077c0fb2e0 2024-02-10T06:15:14.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.421+0000 7f0780f3f700 1 -- 172.21.15.89:0/3174389991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077c04bed0 con 0x7f077c0fb2e0 2024-02-10T06:15:14.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.421+0000 7f076affd700 1 --2- 172.21.15.89:0/3174389991 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f07640692a0 0x7f076406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:14.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.421+0000 7f0779d9b700 1 --2- 172.21.15.89:0/3174389991 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f07640692a0 0x7f076406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:14.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.422+0000 7f076affd700 1 -- 172.21.15.89:0/3174389991 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f076c08b170 con 0x7f077c0fb2e0 2024-02-10T06:15:14.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.422+0000 7f0779d9b700 1 --2- 172.21.15.89:0/3174389991 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f07640692a0 0x7f076406b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0770009b30 tx=0x7f0770009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:14.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.426+0000 7f076affd700 1 -- 172.21.15.89:0/3174389991 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f076c08c060 con 0x7f077c0fb2e0 2024-02-10T06:15:14.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.636+0000 7f0780f3f700 1 -- 172.21.15.89:0/3174389991 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f077c100da0 con 0x7f07640692a0 2024-02-10T06:15:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.637+0000 7f076affd700 1 -- 172.21.15.89:0/3174389991 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f077c100da0 con 0x7f07640692a0 2024-02-10T06:15:14.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.640+0000 7f0780f3f700 1 -- 172.21.15.89:0/3174389991 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f07640692a0 msgr2=0x7f076406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:14.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.640+0000 7f0780f3f700 1 --2- 172.21.15.89:0/3174389991 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f07640692a0 0x7f076406b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0770009b30 tx=0x7f0770009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:14.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.640+0000 7f0780f3f700 1 -- 172.21.15.89:0/3174389991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0fb2e0 msgr2=0x7f077c0fde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:14.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.640+0000 7f0780f3f700 1 --2- 172.21.15.89:0/3174389991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0fb2e0 0x7f077c0fde40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f076c00eb40 tx=0x7f076c00b040 comp rx=0 tx=0).stop 2024-02-10T06:15:14.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.640+0000 7f0780f3f700 1 -- 172.21.15.89:0/3174389991 shutdown_connections 2024-02-10T06:15:14.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.640+0000 7f0780f3f700 1 --2- 172.21.15.89:0/3174389991 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f07640692a0 0x7f076406b750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.640+0000 7f0780f3f700 1 --2- 172.21.15.89:0/3174389991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f077c0fb2e0 0x7f077c0fde40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.640+0000 7f0780f3f700 1 --2- 172.21.15.89:0/3174389991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f077c0fc2c0 0x7f077c0fc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.640+0000 7f0780f3f700 1 -- 172.21.15.89:0/3174389991 >> 172.21.15.89:0/3174389991 conn(0x7f077c0f6ca0 msgr2=0x7f077c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:14.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.640+0000 7f0780f3f700 1 -- 172.21.15.89:0/3174389991 shutdown_connections 2024-02-10T06:15:14.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.640+0000 7f0780f3f700 1 -- 172.21.15.89:0/3174389991 wait complete. 2024-02-10T06:15:14.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.742+0000 7fbe6a686700 1 -- 172.21.15.89:0/3715615800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64066c50 msgr2=0x7fbe640ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:14.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.742+0000 7fbe6a686700 1 --2- 172.21.15.89:0/3715615800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64066c50 0x7fbe640ffb20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbe54004940 tx=0x7fbe54009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:14.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.743+0000 7fbe6a686700 1 -- 172.21.15.89:0/3715615800 shutdown_connections 2024-02-10T06:15:14.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.743+0000 7fbe6a686700 1 --2- 172.21.15.89:0/3715615800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe641000f0 0x7fbe64102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.743+0000 7fbe6a686700 1 --2- 172.21.15.89:0/3715615800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe64066c50 0x7fbe640ffb20 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.743+0000 7fbe6a686700 1 -- 172.21.15.89:0/3715615800 >> 172.21.15.89:0/3715615800 conn(0x7fbe640f7510 msgr2=0x7fbe640f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:14.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.743+0000 7fbe6a686700 1 -- 172.21.15.89:0/3715615800 shutdown_connections 2024-02-10T06:15:14.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.743+0000 7fbe6a686700 1 -- 172.21.15.89:0/3715615800 wait complete. 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.743+0000 7fbe6a686700 1 Processor -- start 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.744+0000 7fbe6a686700 1 -- start start 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.744+0000 7fbe6a686700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe64066c50 0x7fbe6410d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.744+0000 7fbe6a686700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe641000f0 0x7fbe6410deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.744+0000 7fbe6a686700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6410e4b0 con 0x7fbe641000f0 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.744+0000 7fbe6a686700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6410ed50 con 0x7fbe64066c50 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.744+0000 7fbe63fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe64066c50 0x7fbe6410d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.744+0000 7fbe63fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe64066c50 0x7fbe6410d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.89:43302/0 (socket says 172.21.15.89:43302) 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.744+0000 7fbe63fff700 1 -- 172.21.15.89:0/907781899 learned_addr learned my addr 172.21.15.89:0/907781899 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.744+0000 7fbe637fe700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe641000f0 0x7fbe6410deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.745+0000 7fbe63fff700 1 -- 172.21.15.89:0/907781899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe641000f0 msgr2=0x7fbe6410deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:14.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.745+0000 7fbe63fff700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe641000f0 0x7fbe6410deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.745+0000 7fbe63fff700 1 -- 172.21.15.89:0/907781899 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe54005040 con 0x7fbe64066c50 2024-02-10T06:15:14.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.745+0000 7fbe637fe700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe641000f0 0x7fbe6410deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:15:14.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.745+0000 7fbe63fff700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe64066c50 0x7fbe6410d970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbe540093b0 tx=0x7fbe54009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:14.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.745+0000 7fbe60ff9700 1 -- 172.21.15.89:0/907781899 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe5400e040 con 0x7fbe64066c50 2024-02-10T06:15:14.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.745+0000 7fbe6a686700 1 -- 172.21.15.89:0/907781899 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6410efd0 con 0x7fbe64066c50 2024-02-10T06:15:14.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.745+0000 7fbe60ff9700 1 -- 172.21.15.89:0/907781899 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe5400bcc0 con 0x7fbe64066c50 2024-02-10T06:15:14.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.745+0000 7fbe6a686700 1 -- 172.21.15.89:0/907781899 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe6410f510 con 0x7fbe64066c50 2024-02-10T06:15:14.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.746+0000 7fbe60ff9700 1 -- 172.21.15.89:0/907781899 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe5401c630 con 0x7fbe64066c50 2024-02-10T06:15:14.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.746+0000 7fbe6a686700 1 -- 172.21.15.89:0/907781899 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6404bed0 con 0x7fbe64066c50 2024-02-10T06:15:14.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.747+0000 7fbe60ff9700 1 -- 172.21.15.89:0/907781899 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fbe540139f0 con 0x7fbe64066c50 2024-02-10T06:15:14.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.748+0000 7fbe60ff9700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbe4c0692a0 0x7fbe4c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:14.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.748+0000 7fbe60ff9700 1 -- 172.21.15.89:0/907781899 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fbe540884f0 con 0x7fbe64066c50 2024-02-10T06:15:14.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.748+0000 7fbe637fe700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbe4c0692a0 0x7fbe4c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:14.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.749+0000 7fbe637fe700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbe4c0692a0 0x7fbe4c06b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbe64066990 tx=0x7fbe5800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:14.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.756+0000 7fbe60ff9700 1 -- 172.21.15.89:0/907781899 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbe540582f0 con 0x7fbe64066c50 2024-02-10T06:15:14.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.955+0000 7fbe6a686700 1 -- 172.21.15.89:0/907781899 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe64100d50 con 0x7fbe4c0692a0 2024-02-10T06:15:14.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.972+0000 7fbe60ff9700 1 -- 172.21.15.89:0/907781899 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fbe64100d50 con 0x7fbe4c0692a0 2024-02-10T06:15:14.973 INFO:teuthology.orchestra.run.smithi089.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:15:14.973 INFO:teuthology.orchestra.run.smithi089.stdout:alertmanager.smithi089 smithi089 *:9093,9094 running (4m) 28s ago 6m 24.6M - 0.25.0 c8568f914cd2 667ad25cb367 2024-02-10T06:15:14.973 INFO:teuthology.orchestra.run.smithi089.stdout:crash.smithi089 smithi089 running (6m) 28s ago 6m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 e3aac8b18020 2024-02-10T06:15:14.973 INFO:teuthology.orchestra.run.smithi089.stdout:crash.smithi173 smithi173 running (5m) 33s ago 5m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 269616f4b479 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:grafana.smithi089 smithi089 *:3000 running (4m) 28s ago 5m 86.1M - 9.4.7 954c08fa6188 a6f72269ee53 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi089.evydlq smithi089 running (46s) 28s ago 45s 14.2M - 17.2.7-505-g72ce7c1d a2a0bd126c96 9e2f6f4766e6 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi089.mgkvmj smithi089 running (53s) 28s ago 52s 10.4M - 17.2.7-505-g72ce7c1d a2a0bd126c96 689a4ca9f11b 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi173.cjkmdf smithi173 running (43s) 33s ago 43s 12.0M - 17.2.7-505-g72ce7c1d a2a0bd126c96 3d9eb72f6f12 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi173.tumnkz smithi173 running (50s) 33s ago 50s 17.1M - 17.2.7-505-g72ce7c1d a2a0bd126c96 272e93861538 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.smithi089.pwmrye smithi089 *:9283 running (7m) 28s ago 7m 481M - 17.2.7-505-g72ce7c1d a2a0bd126c96 1cd3521b2ebb 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.smithi173.swdirm smithi173 *:8443,9283 running (5m) 33s ago 5m 433M - 17.2.7-505-g72ce7c1d a2a0bd126c96 6430108d8d2e 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:mon.smithi089 smithi089 running (7m) 28s ago 7m 46.7M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 0109d6119759 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:mon.smithi173 smithi173 running (5m) 33s ago 5m 35.8M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 08af8aefe37b 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:node-exporter.smithi089 smithi089 *:9100 running (6m) 28s ago 6m 14.7M - 1.5.0 0da6a335fe13 18b1f6a49022 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:node-exporter.smithi173 smithi173 *:9100 running (5m) 33s ago 5m 16.5M - 1.5.0 0da6a335fe13 f7d0b906813f 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0 smithi089 running (4m) 28s ago 4m 47.5M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 e372a21351db 2024-02-10T06:15:14.974 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1 smithi089 running (3m) 28s ago 3m 46.3M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 42f862e2f6ca 2024-02-10T06:15:14.975 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2 smithi089 running (3m) 28s ago 3m 45.2M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 69b16e6f89fb 2024-02-10T06:15:14.975 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3 smithi173 running (3m) 33s ago 3m 43.3M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 997c2081faec 2024-02-10T06:15:14.975 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4 smithi173 running (2m) 33s ago 2m 42.0M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 d3c6d0e77356 2024-02-10T06:15:14.975 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5 smithi173 running (2m) 33s ago 2m 41.2M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 bb845c14ca98 2024-02-10T06:15:14.975 INFO:teuthology.orchestra.run.smithi089.stdout:prometheus.smithi089 smithi089 *:9095 running (4m) 28s ago 5m 53.8M - 2.43.0 a07b618ecd1d 04539fe88109 2024-02-10T06:15:14.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.974+0000 7fbe6a686700 1 -- 172.21.15.89:0/907781899 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbe4c0692a0 msgr2=0x7fbe4c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:14.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.974+0000 7fbe6a686700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbe4c0692a0 0x7fbe4c06b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbe64066990 tx=0x7fbe5800b040 comp rx=0 tx=0).stop 2024-02-10T06:15:14.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.974+0000 7fbe6a686700 1 -- 172.21.15.89:0/907781899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe64066c50 msgr2=0x7fbe6410d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:14.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.974+0000 7fbe6a686700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe64066c50 0x7fbe6410d970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbe540093b0 tx=0x7fbe54009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:14.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.974+0000 7fbe6a686700 1 -- 172.21.15.89:0/907781899 shutdown_connections 2024-02-10T06:15:14.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.974+0000 7fbe6a686700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe64066c50 0x7fbe6410d970 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.974+0000 7fbe6a686700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbe4c0692a0 0x7fbe4c06b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.974+0000 7fbe6a686700 1 --2- 172.21.15.89:0/907781899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe641000f0 0x7fbe6410deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:14.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.974+0000 7fbe6a686700 1 -- 172.21.15.89:0/907781899 >> 172.21.15.89:0/907781899 conn(0x7fbe640f7510 msgr2=0x7fbe640f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:14.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.974+0000 7fbe6a686700 1 -- 172.21.15.89:0/907781899 shutdown_connections 2024-02-10T06:15:14.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:14.974+0000 7fbe6a686700 1 -- 172.21.15.89:0/907781899 wait complete. 2024-02-10T06:15:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.077+0000 7f3c0da10700 1 -- 172.21.15.89:0/3219981563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c080fda80 msgr2=0x7f3c080ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.077+0000 7f3c0da10700 1 --2- 172.21.15.89:0/3219981563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c080fda80 0x7f3c080ffe60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8004940 tx=0x7f3bf8009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.077+0000 7f3c0da10700 1 -- 172.21.15.89:0/3219981563 shutdown_connections 2024-02-10T06:15:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.077+0000 7f3c0da10700 1 --2- 172.21.15.89:0/3219981563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c081003a0 0x7f3c08102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.077+0000 7f3c0da10700 1 --2- 172.21.15.89:0/3219981563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c080fda80 0x7f3c080ffe60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:15.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.077+0000 7f3c0da10700 1 -- 172.21.15.89:0/3219981563 >> 172.21.15.89:0/3219981563 conn(0x7f3c080f7500 msgr2=0x7f3c080f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:15.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.077+0000 7f3c0da10700 1 -- 172.21.15.89:0/3219981563 shutdown_connections 2024-02-10T06:15:15.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.077+0000 7f3c0da10700 1 -- 172.21.15.89:0/3219981563 wait complete. 2024-02-10T06:15:15.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.077+0000 7f3c0da10700 1 Processor -- start 2024-02-10T06:15:15.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.077+0000 7f3c0da10700 1 -- start start 2024-02-10T06:15:15.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c0da10700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c080fda80 0x7f3c080fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:15.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c0da10700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c081003a0 0x7f3c080fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:15.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c0da10700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c08111570 con 0x7f3c080fda80 2024-02-10T06:15:15.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c0da10700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c081116e0 con 0x7f3c081003a0 2024-02-10T06:15:15.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c06ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c080fda80 0x7f3c080fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:15.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c06ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c080fda80 0x7f3c080fd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37262/0 (socket says 172.21.15.89:37262) 2024-02-10T06:15:15.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c06ffd700 1 -- 172.21.15.89:0/655879773 learned_addr learned my addr 172.21.15.89:0/655879773 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:15.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c067fc700 1 --2- 172.21.15.89:0/655879773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c081003a0 0x7f3c080fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:15.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c06ffd700 1 -- 172.21.15.89:0/655879773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c081003a0 msgr2=0x7f3c080fb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:15.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c06ffd700 1 --2- 172.21.15.89:0/655879773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c081003a0 0x7f3c080fb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:15.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c06ffd700 1 -- 172.21.15.89:0/655879773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf8005040 con 0x7f3c080fda80 2024-02-10T06:15:15.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.078+0000 7f3c06ffd700 1 --2- 172.21.15.89:0/655879773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c080fda80 0x7f3c080fd510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8000c00 tx=0x7f3bf8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:15.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.079+0000 7f3bef7fe700 1 -- 172.21.15.89:0/655879773 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bf8021070 con 0x7f3c080fda80 2024-02-10T06:15:15.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.079+0000 7f3bef7fe700 1 -- 172.21.15.89:0/655879773 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3bf801c400 con 0x7f3c080fda80 2024-02-10T06:15:15.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.080+0000 7f3bef7fe700 1 -- 172.21.15.89:0/655879773 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bf800e070 con 0x7f3c080fda80 2024-02-10T06:15:15.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.080+0000 7f3c0da10700 1 -- 172.21.15.89:0/655879773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c080fbf90 con 0x7f3c080fda80 2024-02-10T06:15:15.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.080+0000 7f3c0da10700 1 -- 172.21.15.89:0/655879773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c080fc440 con 0x7f3c080fda80 2024-02-10T06:15:15.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.081+0000 7f3c0da10700 1 -- 172.21.15.89:0/655879773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c08107a50 con 0x7f3c080fda80 2024-02-10T06:15:15.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.082+0000 7f3bef7fe700 1 -- 172.21.15.89:0/655879773 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3bf801ca50 con 0x7f3c080fda80 2024-02-10T06:15:15.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.083+0000 7f3bef7fe700 1 --2- 172.21.15.89:0/655879773 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f3bf0069360 0x7f3bf006b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:15.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.083+0000 7f3bef7fe700 1 -- 172.21.15.89:0/655879773 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3bf8087cb0 con 0x7f3c080fda80 2024-02-10T06:15:15.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.083+0000 7f3c067fc700 1 --2- 172.21.15.89:0/655879773 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f3bf0069360 0x7f3bf006b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:15.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.084+0000 7f3c067fc700 1 --2- 172.21.15.89:0/655879773 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f3bf0069360 0x7f3bf006b810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009b80 tx=0x7f3bfc00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:15.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.087+0000 7f3bef7fe700 1 -- 172.21.15.89:0/655879773 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3bf8057ab0 con 0x7f3c080fda80 2024-02-10T06:15:15.220 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:15 smithi089 ceph-mon[29217]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-10T06:15:15.220 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:15 smithi089 ceph-mon[29217]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:15.220 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:15 smithi089 ceph-mon[29217]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:15.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.491+0000 7f3c0da10700 1 -- 172.21.15.89:0/655879773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3c080fc750 con 0x7f3c080fda80 2024-02-10T06:15:15.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.492+0000 7f3bef7fe700 1 -- 172.21.15.89:0/655879773 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f3bf8057640 con 0x7f3c080fda80 2024-02-10T06:15:15.493 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-10T06:15:15.493 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": { 2024-02-10T06:15:15.493 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-10T06:15:15.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:15:15.493 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": { 2024-02-10T06:15:15.493 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-10T06:15:15.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:15:15.493 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": { 2024-02-10T06:15:15.493 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 6 2024-02-10T06:15:15.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:15:15.494 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": { 2024-02-10T06:15:15.494 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 4 2024-02-10T06:15:15.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:15:15.494 INFO:teuthology.orchestra.run.smithi089.stdout: "overall": { 2024-02-10T06:15:15.494 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 14 2024-02-10T06:15:15.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-10T06:15:15.494 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-10T06:15:15.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.495+0000 7f3c0da10700 1 -- 172.21.15.89:0/655879773 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f3bf0069360 msgr2=0x7f3bf006b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:15.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.495+0000 7f3c0da10700 1 --2- 172.21.15.89:0/655879773 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f3bf0069360 0x7f3bf006b810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009b80 tx=0x7f3bfc00b040 comp rx=0 tx=0).stop 2024-02-10T06:15:15.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.495+0000 7f3c0da10700 1 -- 172.21.15.89:0/655879773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c080fda80 msgr2=0x7f3c080fd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:15.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.495+0000 7f3c0da10700 1 --2- 172.21.15.89:0/655879773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c080fda80 0x7f3c080fd510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8000c00 tx=0x7f3bf8009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:15.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.496+0000 7f3c0da10700 1 -- 172.21.15.89:0/655879773 shutdown_connections 2024-02-10T06:15:15.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.496+0000 7f3c0da10700 1 --2- 172.21.15.89:0/655879773 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f3bf0069360 0x7f3bf006b810 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:15.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.496+0000 7f3c0da10700 1 --2- 172.21.15.89:0/655879773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c080fda80 0x7f3c080fd510 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:15.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.496+0000 7f3c0da10700 1 --2- 172.21.15.89:0/655879773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c081003a0 0x7f3c080fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:15.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.496+0000 7f3c0da10700 1 -- 172.21.15.89:0/655879773 >> 172.21.15.89:0/655879773 conn(0x7f3c080f7500 msgr2=0x7f3c080f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:15.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.496+0000 7f3c0da10700 1 -- 172.21.15.89:0/655879773 shutdown_connections 2024-02-10T06:15:15.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.496+0000 7f3c0da10700 1 -- 172.21.15.89:0/655879773 wait complete. 2024-02-10T06:15:15.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.637+0000 7f99c3710700 1 -- 172.21.15.89:0/151200075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99bc0fe0e0 msgr2=0x7f99bc0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:15.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.637+0000 7f99c3710700 1 --2- 172.21.15.89:0/151200075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99bc0fe0e0 0x7f99bc0fe550 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f99ac004940 tx=0x7f99ac009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:15.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.637+0000 7f99c3710700 1 -- 172.21.15.89:0/151200075 shutdown_connections 2024-02-10T06:15:15.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.637+0000 7f99c3710700 1 --2- 172.21.15.89:0/151200075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99bc0fe0e0 0x7f99bc0fe550 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:15.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.637+0000 7f99c3710700 1 --2- 172.21.15.89:0/151200075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99bc0fcef0 0x7f99bc0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:15.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.637+0000 7f99c3710700 1 -- 172.21.15.89:0/151200075 >> 172.21.15.89:0/151200075 conn(0x7f99bc0f8620 msgr2=0x7f99bc0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:15.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.637+0000 7f99c3710700 1 -- 172.21.15.89:0/151200075 shutdown_connections 2024-02-10T06:15:15.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.637+0000 7f99c3710700 1 -- 172.21.15.89:0/151200075 wait complete. 2024-02-10T06:15:15.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.638+0000 7f99c3710700 1 Processor -- start 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.638+0000 7f99c3710700 1 -- start start 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.638+0000 7f99c3710700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99bc0fcef0 0x7f99bc071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.638+0000 7f99c3710700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99bc0fe0e0 0x7f99bc0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.638+0000 7f99c3710700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc1115a0 con 0x7f99bc0fcef0 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.638+0000 7f99c3710700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc111710 con 0x7f99bc0fe0e0 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.638+0000 7f99c14ac700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99bc0fcef0 0x7f99bc071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.638+0000 7f99c14ac700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99bc0fcef0 0x7f99bc071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37288/0 (socket says 172.21.15.89:37288) 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.638+0000 7f99c14ac700 1 -- 172.21.15.89:0/1209933092 learned_addr learned my addr 172.21.15.89:0/1209933092 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.638+0000 7f99c0cab700 1 --2- 172.21.15.89:0/1209933092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99bc0fe0e0 0x7f99bc0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.639+0000 7f99c14ac700 1 -- 172.21.15.89:0/1209933092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99bc0fe0e0 msgr2=0x7f99bc0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.639+0000 7f99c14ac700 1 --2- 172.21.15.89:0/1209933092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99bc0fe0e0 0x7f99bc0702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.639+0000 7f99c14ac700 1 -- 172.21.15.89:0/1209933092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac005040 con 0x7f99bc0fcef0 2024-02-10T06:15:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.639+0000 7f99c14ac700 1 --2- 172.21.15.89:0/1209933092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99bc0fcef0 0x7f99bc071e60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f99b800eb40 tx=0x7f99b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:15.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.639+0000 7f99b1ffb700 1 -- 172.21.15.89:0/1209933092 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b8007cc0 con 0x7f99bc0fcef0 2024-02-10T06:15:15.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.639+0000 7f99b1ffb700 1 -- 172.21.15.89:0/1209933092 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99b800f040 con 0x7f99bc0fcef0 2024-02-10T06:15:15.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.639+0000 7f99b1ffb700 1 -- 172.21.15.89:0/1209933092 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b800caa0 con 0x7f99bc0fcef0 2024-02-10T06:15:15.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.639+0000 7f99c3710700 1 -- 172.21.15.89:0/1209933092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99bc0708e0 con 0x7f99bc0fcef0 2024-02-10T06:15:15.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.640+0000 7f99c3710700 1 -- 172.21.15.89:0/1209933092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99bc070e20 con 0x7f99bc0fcef0 2024-02-10T06:15:15.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.646+0000 7f99b1ffb700 1 -- 172.21.15.89:0/1209933092 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f99b800cc00 con 0x7f99bc0fcef0 2024-02-10T06:15:15.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.646+0000 7f99c3710700 1 -- 172.21.15.89:0/1209933092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99bc04bed0 con 0x7f99bc0fcef0 2024-02-10T06:15:15.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.647+0000 7f99b1ffb700 1 --2- 172.21.15.89:0/1209933092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99a80692a0 0x7f99a806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:15.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.647+0000 7f99c0cab700 1 --2- 172.21.15.89:0/1209933092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99a80692a0 0x7f99a806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:15.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.648+0000 7f99b1ffb700 1 -- 172.21.15.89:0/1209933092 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f99b8007e20 con 0x7f99bc0fcef0 2024-02-10T06:15:15.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.648+0000 7f99c0cab700 1 --2- 172.21.15.89:0/1209933092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99a80692a0 0x7f99a806b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00b0d0 tx=0x7f99ac009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:15.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:15 smithi173 ceph-mon[36251]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-10T06:15:15.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:15 smithi173 ceph-mon[36251]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:15.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:15 smithi173 ceph-mon[36251]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:15.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.656+0000 7f99b1ffb700 1 -- 172.21.15.89:0/1209933092 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f99b808c060 con 0x7f99bc0fcef0 2024-02-10T06:15:15.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.983+0000 7f99c3710700 1 -- 172.21.15.89:0/1209933092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f99bc105a50 con 0x7f99bc0fcef0 2024-02-10T06:15:15.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.983+0000 7f99b1ffb700 1 -- 172.21.15.89:0/1209933092 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1838 (secure 0 0 0) 0x7f99b8019740 con 0x7f99bc0fcef0 2024-02-10T06:15:15.986 INFO:teuthology.orchestra.run.smithi089.stdout:e15 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:legacy client fscid: 1 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:Filesystem 'cephfs' (1) 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:fs_name cephfs 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 15 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-02-10T06:14:18.236041+0000 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:modified 2024-02-10T06:14:36.018945+0000 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:tableserver 0 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:root 0 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:session_timeout 60 2024-02-10T06:15:15.987 INFO:teuthology.orchestra.run.smithi089.stdout:session_autoclose 300 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:max_file_size 1099511627776 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:required_client_features {} 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure 0 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure_osd_epoch 47 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:max_mds 1 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:in 0 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:up {0=24235} 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:failed 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:damaged 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:stopped 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:data_pools [3] 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:metadata_pool 2 2024-02-10T06:15:15.988 INFO:teuthology.orchestra.run.smithi089.stdout:inline_data enabled 2024-02-10T06:15:15.989 INFO:teuthology.orchestra.run.smithi089.stdout:balancer 2024-02-10T06:15:15.989 INFO:teuthology.orchestra.run.smithi089.stdout:standby_count_wanted 1 2024-02-10T06:15:15.989 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi173.tumnkz{0:24235} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:15:15.989 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi089.evydlq{0:14476} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.89:6828/3386828748,v1:172.21.15.89:6829/3386828748] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:15:15.989 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:15:15.989 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:15:15.989 INFO:teuthology.orchestra.run.smithi089.stdout:Standby daemons: 2024-02-10T06:15:15.989 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:15:15.989 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi173.cjkmdf{-1:24243} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.173:6826/2798078299,v1:172.21.15.173:6827/2798078299] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:15:15.989 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi089.mgkvmj{-1:24251} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.89:6826/3574254294,v1:172.21.15.89:6827/3574254294] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:15:15.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.987+0000 7f99c3710700 1 -- 172.21.15.89:0/1209933092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99a80692a0 msgr2=0x7f99a806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:15.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.987+0000 7f99c3710700 1 --2- 172.21.15.89:0/1209933092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99a80692a0 0x7f99a806b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00b0d0 tx=0x7f99ac009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:15.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.987+0000 7f99c3710700 1 -- 172.21.15.89:0/1209933092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99bc0fcef0 msgr2=0x7f99bc071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:15.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.987+0000 7f99c3710700 1 --2- 172.21.15.89:0/1209933092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99bc0fcef0 0x7f99bc071e60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f99b800eb40 tx=0x7f99b800b040 comp rx=0 tx=0).stop 2024-02-10T06:15:15.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.987+0000 7f99c3710700 1 -- 172.21.15.89:0/1209933092 shutdown_connections 2024-02-10T06:15:15.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.987+0000 7f99c3710700 1 --2- 172.21.15.89:0/1209933092 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f99a80692a0 0x7f99a806b750 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00b0d0 tx=0x7f99ac009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:15.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.987+0000 7f99c3710700 1 --2- 172.21.15.89:0/1209933092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99bc0fcef0 0x7f99bc071e60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:15.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.987+0000 7f99c3710700 1 --2- 172.21.15.89:0/1209933092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99bc0fe0e0 0x7f99bc0702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:15.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.987+0000 7f99c3710700 1 -- 172.21.15.89:0/1209933092 >> 172.21.15.89:0/1209933092 conn(0x7f99bc0f8620 msgr2=0x7f99bc1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:15.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.988+0000 7f99c3710700 1 -- 172.21.15.89:0/1209933092 shutdown_connections 2024-02-10T06:15:15.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:15.989+0000 7f99c3710700 1 -- 172.21.15.89:0/1209933092 wait complete. 2024-02-10T06:15:15.992 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 15 2024-02-10T06:15:16.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.112+0000 7fe6951f9700 1 -- 172.21.15.89:0/1488100911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6900fcfb0 msgr2=0x7fe6900fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:16.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.112+0000 7fe6951f9700 1 --2- 172.21.15.89:0/1488100911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6900fcfb0 0x7fe6900fd420 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe680004940 tx=0x7fe680009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:16.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.112+0000 7fe6951f9700 1 -- 172.21.15.89:0/1488100911 shutdown_connections 2024-02-10T06:15:16.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.112+0000 7fe6951f9700 1 --2- 172.21.15.89:0/1488100911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6900fcfb0 0x7fe6900fd420 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:16.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.112+0000 7fe6951f9700 1 --2- 172.21.15.89:0/1488100911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6900fbe40 0x7fe6900fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:16.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.112+0000 7fe6951f9700 1 -- 172.21.15.89:0/1488100911 >> 172.21.15.89:0/1488100911 conn(0x7fe6900f74e0 msgr2=0x7fe6900f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:16.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.112+0000 7fe6951f9700 1 -- 172.21.15.89:0/1488100911 shutdown_connections 2024-02-10T06:15:16.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.112+0000 7fe6951f9700 1 -- 172.21.15.89:0/1488100911 wait complete. 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.113+0000 7fe6951f9700 1 Processor -- start 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.113+0000 7fe6951f9700 1 -- start start 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.113+0000 7fe6951f9700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6900fbe40 0x7fe69010c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.113+0000 7fe6951f9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6900fcfb0 0x7fe69010cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.113+0000 7fe6951f9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69010d310 con 0x7fe6900fcfb0 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.113+0000 7fe6951f9700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69010dbb0 con 0x7fe6900fbe40 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.113+0000 7fe68ed9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6900fbe40 0x7fe69010c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.113+0000 7fe68ed9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6900fbe40 0x7fe69010c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.89:43348/0 (socket says 172.21.15.89:43348) 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.113+0000 7fe68e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6900fcfb0 0x7fe69010cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.113+0000 7fe68ed9d700 1 -- 172.21.15.89:0/1201027787 learned_addr learned my addr 172.21.15.89:0/1201027787 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.114+0000 7fe68ed9d700 1 -- 172.21.15.89:0/1201027787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6900fcfb0 msgr2=0x7fe69010cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:16.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.114+0000 7fe68ed9d700 1 --2- 172.21.15.89:0/1201027787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6900fcfb0 0x7fe69010cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:16.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.114+0000 7fe68ed9d700 1 -- 172.21.15.89:0/1201027787 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe680005040 con 0x7fe6900fbe40 2024-02-10T06:15:16.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.114+0000 7fe68ed9d700 1 --2- 172.21.15.89:0/1201027787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6900fbe40 0x7fe69010c7d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe68400eb40 tx=0x7fe68400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:16.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.114+0000 7fe6777fe700 1 -- 172.21.15.89:0/1201027787 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe684007ea0 con 0x7fe6900fbe40 2024-02-10T06:15:16.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.114+0000 7fe6951f9700 1 -- 172.21.15.89:0/1201027787 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69010de30 con 0x7fe6900fbe40 2024-02-10T06:15:16.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.114+0000 7fe6777fe700 1 -- 172.21.15.89:0/1201027787 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6840095a0 con 0x7fe6900fbe40 2024-02-10T06:15:16.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.115+0000 7fe6951f9700 1 -- 172.21.15.89:0/1201027787 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69010e2b0 con 0x7fe6900fbe40 2024-02-10T06:15:16.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.115+0000 7fe6777fe700 1 -- 172.21.15.89:0/1201027787 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe68400dd50 con 0x7fe6900fbe40 2024-02-10T06:15:16.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.116+0000 7fe6951f9700 1 -- 172.21.15.89:0/1201027787 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69004bed0 con 0x7fe6900fbe40 2024-02-10T06:15:16.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.116+0000 7fe6777fe700 1 -- 172.21.15.89:0/1201027787 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe684009710 con 0x7fe6900fbe40 2024-02-10T06:15:16.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.117+0000 7fe6777fe700 1 --2- 172.21.15.89:0/1201027787 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe678069380 0x7fe67806b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:16.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.117+0000 7fe6777fe700 1 -- 172.21.15.89:0/1201027787 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe684015020 con 0x7fe6900fbe40 2024-02-10T06:15:16.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.117+0000 7fe68e59c700 1 --2- 172.21.15.89:0/1201027787 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe678069380 0x7fe67806b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:16.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.117+0000 7fe68e59c700 1 --2- 172.21.15.89:0/1201027787 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe678069380 0x7fe67806b830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe680013740 tx=0x7fe68001c040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:16.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.125+0000 7fe6777fe700 1 -- 172.21.15.89:0/1201027787 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe684057870 con 0x7fe6900fbe40 2024-02-10T06:15:16.356 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:16 smithi089 ceph-mon[29217]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:16.357 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:16 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/655879773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:15:16.357 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:16 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1209933092' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:15:16.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.410+0000 7fe6951f9700 1 -- 172.21.15.89:0/1201027787 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe690101990 con 0x7fe678069380 2024-02-10T06:15:16.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.412+0000 7fe6777fe700 1 -- 172.21.15.89:0/1201027787 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fe690101990 con 0x7fe678069380 2024-02-10T06:15:16.413 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-10T06:15:16.413 INFO:teuthology.orchestra.run.smithi089.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T06:15:16.413 INFO:teuthology.orchestra.run.smithi089.stdout: "in_progress": true, 2024-02-10T06:15:16.413 INFO:teuthology.orchestra.run.smithi089.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:15:16.413 INFO:teuthology.orchestra.run.smithi089.stdout: "services_complete": [], 2024-02-10T06:15:16.413 INFO:teuthology.orchestra.run.smithi089.stdout: "progress": "", 2024-02-10T06:15:16.413 INFO:teuthology.orchestra.run.smithi089.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 image", 2024-02-10T06:15:16.413 INFO:teuthology.orchestra.run.smithi089.stdout: "is_paused": false 2024-02-10T06:15:16.413 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-10T06:15:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.414+0000 7fe6951f9700 1 -- 172.21.15.89:0/1201027787 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe678069380 msgr2=0x7fe67806b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.414+0000 7fe6951f9700 1 --2- 172.21.15.89:0/1201027787 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe678069380 0x7fe67806b830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe680013740 tx=0x7fe68001c040 comp rx=0 tx=0).stop 2024-02-10T06:15:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.415+0000 7fe6951f9700 1 -- 172.21.15.89:0/1201027787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6900fbe40 msgr2=0x7fe69010c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.415+0000 7fe6951f9700 1 --2- 172.21.15.89:0/1201027787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6900fbe40 0x7fe69010c7d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe68400eb40 tx=0x7fe68400b040 comp rx=0 tx=0).stop 2024-02-10T06:15:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.415+0000 7fe6951f9700 1 -- 172.21.15.89:0/1201027787 shutdown_connections 2024-02-10T06:15:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.415+0000 7fe6951f9700 1 --2- 172.21.15.89:0/1201027787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6900fbe40 0x7fe69010c7d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.415+0000 7fe6951f9700 1 --2- 172.21.15.89:0/1201027787 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe678069380 0x7fe67806b830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.415+0000 7fe6951f9700 1 --2- 172.21.15.89:0/1201027787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6900fcfb0 0x7fe69010cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.415+0000 7fe6951f9700 1 -- 172.21.15.89:0/1201027787 >> 172.21.15.89:0/1201027787 conn(0x7fe6900f74e0 msgr2=0x7fe690100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.415+0000 7fe6951f9700 1 -- 172.21.15.89:0/1201027787 shutdown_connections 2024-02-10T06:15:16.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.415+0000 7fe6951f9700 1 -- 172.21.15.89:0/1201027787 wait complete. 2024-02-10T06:15:16.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.552+0000 7fe05e7a6700 1 -- 172.21.15.89:0/3868049514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0580fcef0 msgr2=0x7fe0580fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:16.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.552+0000 7fe05e7a6700 1 --2- 172.21.15.89:0/3868049514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0580fcef0 0x7fe0580fd320 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe048004940 tx=0x7fe048009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:16.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.552+0000 7fe05e7a6700 1 -- 172.21.15.89:0/3868049514 shutdown_connections 2024-02-10T06:15:16.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.552+0000 7fe05e7a6700 1 --2- 172.21.15.89:0/3868049514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0580fe0e0 0x7fe0580fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:16.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.552+0000 7fe05e7a6700 1 --2- 172.21.15.89:0/3868049514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0580fcef0 0x7fe0580fd320 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:16.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.552+0000 7fe05e7a6700 1 -- 172.21.15.89:0/3868049514 >> 172.21.15.89:0/3868049514 conn(0x7fe0580f8620 msgr2=0x7fe0580faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:16.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.552+0000 7fe05e7a6700 1 -- 172.21.15.89:0/3868049514 shutdown_connections 2024-02-10T06:15:16.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.552+0000 7fe05e7a6700 1 -- 172.21.15.89:0/3868049514 wait complete. 2024-02-10T06:15:16.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.554+0000 7fe05e7a6700 1 Processor -- start 2024-02-10T06:15:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.555+0000 7fe05e7a6700 1 -- start start 2024-02-10T06:15:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.555+0000 7fe05e7a6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0580fe0e0 0x7fe05810dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.555+0000 7fe05e7a6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe05810e010 0x7fe05810eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.555+0000 7fe057fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0580fe0e0 0x7fe05810dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.555+0000 7fe057fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0580fe0e0 0x7fe05810dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37312/0 (socket says 172.21.15.89:37312) 2024-02-10T06:15:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.555+0000 7fe057fff700 1 -- 172.21.15.89:0/54751653 learned_addr learned my addr 172.21.15.89:0/54751653 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.555+0000 7fe0577fe700 1 --2- 172.21.15.89:0/54751653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe05810e010 0x7fe05810eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.555+0000 7fe05e7a6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05810e4d0 con 0x7fe0580fe0e0 2024-02-10T06:15:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.556+0000 7fe05e7a6700 1 -- 172.21.15.89:0/54751653 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05810e610 con 0x7fe05810e010 2024-02-10T06:15:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.556+0000 7fe057fff700 1 -- 172.21.15.89:0/54751653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe05810e010 msgr2=0x7fe05810eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.556+0000 7fe057fff700 1 --2- 172.21.15.89:0/54751653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe05810e010 0x7fe05810eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.556+0000 7fe057fff700 1 -- 172.21.15.89:0/54751653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe048005040 con 0x7fe0580fe0e0 2024-02-10T06:15:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.556+0000 7fe057fff700 1 --2- 172.21.15.89:0/54751653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0580fe0e0 0x7fe05810dad0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe048000c00 tx=0x7fe048009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.556+0000 7fe054ff9700 1 -- 172.21.15.89:0/54751653 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe04800e070 con 0x7fe0580fe0e0 2024-02-10T06:15:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.557+0000 7fe054ff9700 1 -- 172.21.15.89:0/54751653 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe048013de0 con 0x7fe0580fe0e0 2024-02-10T06:15:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.557+0000 7fe054ff9700 1 -- 172.21.15.89:0/54751653 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe04801ce70 con 0x7fe0580fe0e0 2024-02-10T06:15:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.557+0000 7fe05e7a6700 1 -- 172.21.15.89:0/54751653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05810f4b0 con 0x7fe0580fe0e0 2024-02-10T06:15:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.557+0000 7fe05e7a6700 1 -- 172.21.15.89:0/54751653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05810f850 con 0x7fe0580fe0e0 2024-02-10T06:15:16.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.559+0000 7fe03e7fc700 1 -- 172.21.15.89:0/54751653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05804bed0 con 0x7fe0580fe0e0 2024-02-10T06:15:16.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.559+0000 7fe054ff9700 1 -- 172.21.15.89:0/54751653 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe04801c400 con 0x7fe0580fe0e0 2024-02-10T06:15:16.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.560+0000 7fe054ff9700 1 --2- 172.21.15.89:0/54751653 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe0400692b0 0x7fe04006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:16.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.560+0000 7fe054ff9700 1 -- 172.21.15.89:0/54751653 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe0480887b0 con 0x7fe0580fe0e0 2024-02-10T06:15:16.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.561+0000 7fe0577fe700 1 --2- 172.21.15.89:0/54751653 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe0400692b0 0x7fe04006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:16.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.561+0000 7fe0577fe700 1 --2- 172.21.15.89:0/54751653 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe0400692b0 0x7fe04006b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe04c009b80 tx=0x7fe04c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:16.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:16.565+0000 7fe054ff9700 1 -- 172.21.15.89:0/54751653 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe048058500 con 0x7fe0580fe0e0 2024-02-10T06:15:16.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:16 smithi173 ceph-mon[36251]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:16.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:16 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/655879773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:15:16.652 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:16 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/1209933092' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:15:17.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.000+0000 7fe03e7fc700 1 -- 172.21.15.89:0/54751653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe058105a50 con 0x7fe0580fe0e0 2024-02-10T06:15:17.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.000+0000 7fe054ff9700 1 -- 172.21.15.89:0/54751653 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fe048086e60 con 0x7fe0580fe0e0 2024-02-10T06:15:17.001 INFO:teuthology.orchestra.run.smithi089.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-10T06:15:17.001 INFO:teuthology.orchestra.run.smithi089.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-10T06:15:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-10T06:15:17.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.003+0000 7fe03e7fc700 1 -- 172.21.15.89:0/54751653 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe0400692b0 msgr2=0x7fe04006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:17.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.003+0000 7fe03e7fc700 1 --2- 172.21.15.89:0/54751653 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe0400692b0 0x7fe04006b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe04c009b80 tx=0x7fe04c00b040 comp rx=0 tx=0).stop 2024-02-10T06:15:17.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.003+0000 7fe03e7fc700 1 -- 172.21.15.89:0/54751653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0580fe0e0 msgr2=0x7fe05810dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:17.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.003+0000 7fe03e7fc700 1 --2- 172.21.15.89:0/54751653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0580fe0e0 0x7fe05810dad0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe048000c00 tx=0x7fe048009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:17.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.003+0000 7fe03e7fc700 1 -- 172.21.15.89:0/54751653 shutdown_connections 2024-02-10T06:15:17.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.003+0000 7fe03e7fc700 1 --2- 172.21.15.89:0/54751653 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fe0400692b0 0x7fe04006b760 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:17.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.003+0000 7fe03e7fc700 1 --2- 172.21.15.89:0/54751653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0580fe0e0 0x7fe05810dad0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:17.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.003+0000 7fe03e7fc700 1 --2- 172.21.15.89:0/54751653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe05810e010 0x7fe05810eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:17.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.003+0000 7fe03e7fc700 1 -- 172.21.15.89:0/54751653 >> 172.21.15.89:0/54751653 conn(0x7fe0580f8620 msgr2=0x7fe0581013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:17.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.004+0000 7fe03e7fc700 1 -- 172.21.15.89:0/54751653 shutdown_connections 2024-02-10T06:15:17.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:17.004+0000 7fe03e7fc700 1 -- 172.21.15.89:0/54751653 wait complete. 2024-02-10T06:15:17.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:17 smithi089 ceph-mon[29217]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:17.356 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:17 smithi089 ceph-mon[29217]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:17.356 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:17 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/54751653' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:15:17.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:17 smithi173 ceph-mon[36251]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:17.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:17 smithi173 ceph-mon[36251]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:17.653 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:17 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/54751653' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:15:19.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:19 smithi089 ceph-mon[29217]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:19.399 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:19 smithi173 ceph-mon[36251]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:21.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:21 smithi089 ceph-mon[29217]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-10T06:15:21.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:21 smithi173 ceph-mon[36251]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-10T06:15:23.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:23 smithi089 ceph-mon[29217]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:23.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:23 smithi173 ceph-mon[36251]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:25.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:25 smithi089 ceph-mon[29217]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-10T06:15:25.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:25 smithi173 ceph-mon[36251]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-10T06:15:27.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:27 smithi089 ceph-mon[29217]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:27.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:27 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:27.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:27 smithi173 ceph-mon[36251]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:27.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:27 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:29.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:29 smithi089 ceph-mon[29217]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:29.399 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:29 smithi173 ceph-mon[36251]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-10T06:15:31.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:31 smithi089 ceph-mon[29217]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-02-10T06:15:31.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:31 smithi173 ceph-mon[36251]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-02-10T06:15:33.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:33 smithi089 ceph-mon[29217]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-10T06:15:33.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:33 smithi173 ceph-mon[36251]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-10T06:15:35.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:35 smithi089 ceph-mon[29217]: pgmap v211: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-10T06:15:35.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:35 smithi173 ceph-mon[36251]: pgmap v211: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-10T06:15:37.486 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:37 smithi089 ceph-mon[29217]: pgmap v212: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-10T06:15:37.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:37 smithi173 ceph-mon[36251]: pgmap v212: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-10T06:15:39.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:39 smithi089 ceph-mon[29217]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-10T06:15:39.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:39 smithi173 ceph-mon[36251]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-10T06:15:41.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:41 smithi089 ceph-mon[29217]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 682 B/s wr, 3 op/s 2024-02-10T06:15:41.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:41 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:15:41.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:41 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:41.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:41 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:15:41.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:41 smithi173 ceph-mon[36251]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 682 B/s wr, 3 op/s 2024-02-10T06:15:41.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:41 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:15:41.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:41 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:41.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:41 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:15:42.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:42 smithi089 ceph-mon[29217]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-10T06:15:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:42 smithi089 ceph-mon[29217]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:15:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:42 smithi089 ceph-mon[29217]: Upgrade: Need to upgrade myself (mgr.smithi089.pwmrye) 2024-02-10T06:15:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:42 smithi089 ceph-mon[29217]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 on smithi173 2024-02-10T06:15:42.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:42 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:42.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:42 smithi173 ceph-mon[36251]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-10T06:15:42.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:42 smithi173 ceph-mon[36251]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T06:15:42.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:42 smithi173 ceph-mon[36251]: Upgrade: Need to upgrade myself (mgr.smithi089.pwmrye) 2024-02-10T06:15:42.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:42 smithi173 ceph-mon[36251]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 on smithi173 2024-02-10T06:15:42.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:42 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:42.969 INFO:tasks.workunit.client.1.smithi173.stderr:Updating files: 91% (11093/12104) Updating files: 92% (11136/12104) Updating files: 93% (11257/12104) Updating files: 94% (11378/12104) Updating files: 95% (11499/12104) Updating files: 96% (11620/12104) Updating files: 97% (11741/12104) Updating files: 98% (11862/12104) Updating files: 99% (11983/12104) Updating files: 100% (12104/12104) Updating files: 100% (12104/12104), done. 2024-02-10T06:15:43.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:43 smithi089 ceph-mon[29217]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-10T06:15:43.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:43 smithi173 ceph-mon[36251]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-10T06:15:43.903 INFO:tasks.workunit.client.1.smithi173.stderr:Note: switching to '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'. 2024-02-10T06:15:43.903 INFO:tasks.workunit.client.1.smithi173.stderr: 2024-02-10T06:15:43.903 INFO:tasks.workunit.client.1.smithi173.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-10T06:15:43.903 INFO:tasks.workunit.client.1.smithi173.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-10T06:15:43.903 INFO:tasks.workunit.client.1.smithi173.stderr:state without impacting any branches by switching back to a branch. 2024-02-10T06:15:43.903 INFO:tasks.workunit.client.1.smithi173.stderr: 2024-02-10T06:15:43.903 INFO:tasks.workunit.client.1.smithi173.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-10T06:15:43.903 INFO:tasks.workunit.client.1.smithi173.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-10T06:15:43.903 INFO:tasks.workunit.client.1.smithi173.stderr: 2024-02-10T06:15:43.903 INFO:tasks.workunit.client.1.smithi173.stderr: git switch -c 2024-02-10T06:15:43.904 INFO:tasks.workunit.client.1.smithi173.stderr: 2024-02-10T06:15:43.904 INFO:tasks.workunit.client.1.smithi173.stderr:Or undo this operation with: 2024-02-10T06:15:43.904 INFO:tasks.workunit.client.1.smithi173.stderr: 2024-02-10T06:15:43.904 INFO:tasks.workunit.client.1.smithi173.stderr: git switch - 2024-02-10T06:15:43.904 INFO:tasks.workunit.client.1.smithi173.stderr: 2024-02-10T06:15:43.904 INFO:tasks.workunit.client.1.smithi173.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-10T06:15:43.904 INFO:tasks.workunit.client.1.smithi173.stderr: 2024-02-10T06:15:43.904 INFO:tasks.workunit.client.1.smithi173.stderr:HEAD is now at 290212a1bfd Merge pull request #53521 from xxhdx1985126/wip-62857 2024-02-10T06:15:43.910 DEBUG:teuthology.orchestra.run.smithi173:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-02-10T06:15:43.983 INFO:tasks.workunit.client.1.smithi173.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-10T06:15:43.985 INFO:tasks.workunit.client.1.smithi173.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-10T06:15:43.985 INFO:tasks.workunit.client.1.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-10T06:15:44.733 INFO:tasks.workunit.client.1.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-10T06:15:44.791 INFO:tasks.workunit.client.1.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-10T06:15:44.833 INFO:tasks.workunit.client.1.smithi173.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-10T06:15:44.835 INFO:tasks.workunit.client.1.smithi173.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-10T06:15:44.835 INFO:tasks.workunit.client.1.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-10T06:15:44.876 INFO:tasks.workunit.client.1.smithi173.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-10T06:15:44.879 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:15:44.879 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-02-10T06:15:44.936 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-02-10T06:15:44.937 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-10T06:15:44.939 DEBUG:teuthology.orchestra.run.smithi173:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-02-10T06:15:45.000 INFO:tasks.workunit.client.1.smithi173.stderr:+ mkdir -p fsstress 2024-02-10T06:15:45.001 INFO:tasks.workunit.client.1.smithi173.stderr:+ pushd fsstress 2024-02-10T06:15:45.002 INFO:tasks.workunit.client.1.smithi173.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-10T06:15:45.002 INFO:tasks.workunit.client.1.smithi173.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-10T06:15:45.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:45 smithi089 ceph-mon[29217]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-10T06:15:45.631 INFO:tasks.workunit.client.1.smithi173.stderr:+ tar xzf ltp-full.tgz 2024-02-10T06:15:45.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:45 smithi173 ceph-mon[36251]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-10T06:15:47.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.117+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2648311826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc40663c0 msgr2=0x7f9cc40ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:47.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.117+0000 7f9ccaa95700 1 --2- 172.21.15.89:0/2648311826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc40663c0 0x7f9cc40ffb10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8004940 tx=0x7f9cb8009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:47.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.117+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2648311826 shutdown_connections 2024-02-10T06:15:47.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.117+0000 7f9ccaa95700 1 --2- 172.21.15.89:0/2648311826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cc41000e0 0x7f9cc4102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.117+0000 7f9ccaa95700 1 --2- 172.21.15.89:0/2648311826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc40663c0 0x7f9cc40ffb10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.117+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2648311826 >> 172.21.15.89:0/2648311826 conn(0x7f9cc40f74e0 msgr2=0x7f9cc40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:47.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.117+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2648311826 shutdown_connections 2024-02-10T06:15:47.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.117+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2648311826 wait complete. 2024-02-10T06:15:47.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.118+0000 7f9ccaa95700 1 Processor -- start 2024-02-10T06:15:47.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.118+0000 7f9ccaa95700 1 -- start start 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.118+0000 7f9ccaa95700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc40663c0 0x7f9cc410d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.118+0000 7f9ccaa95700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cc41000e0 0x7f9cc410de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.118+0000 7f9ccaa95700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc410e420 con 0x7f9cc40663c0 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.118+0000 7f9ccaa95700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc410ecc0 con 0x7f9cc41000e0 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.118+0000 7f9cc8831700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc40663c0 0x7f9cc410d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.118+0000 7f9cc8831700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc40663c0 0x7f9cc410d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37454/0 (socket says 172.21.15.89:37454) 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.118+0000 7f9cc8831700 1 -- 172.21.15.89:0/2102756438 learned_addr learned my addr 172.21.15.89:0/2102756438 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.118+0000 7f9cc3fff700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cc41000e0 0x7f9cc410de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.119+0000 7f9cc3fff700 1 -- 172.21.15.89:0/2102756438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc40663c0 msgr2=0x7f9cc410d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.119+0000 7f9cc3fff700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc40663c0 0x7f9cc410d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.119+0000 7f9cc3fff700 1 -- 172.21.15.89:0/2102756438 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb8005040 con 0x7f9cc41000e0 2024-02-10T06:15:47.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.119+0000 7f9cc8831700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc40663c0 0x7f9cc410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:15:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.119+0000 7f9cc3fff700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cc41000e0 0x7f9cc410de20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4066100 tx=0x7f9cb000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.119+0000 7f9cc17fa700 1 -- 172.21.15.89:0/2102756438 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cb0004e70 con 0x7f9cc41000e0 2024-02-10T06:15:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.119+0000 7f9cc17fa700 1 -- 172.21.15.89:0/2102756438 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cb000f040 con 0x7f9cc41000e0 2024-02-10T06:15:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.119+0000 7f9cc17fa700 1 -- 172.21.15.89:0/2102756438 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cb000c950 con 0x7f9cc41000e0 2024-02-10T06:15:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.119+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2102756438 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc410ef40 con 0x7f9cc41000e0 2024-02-10T06:15:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.119+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2102756438 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc410f430 con 0x7f9cc41000e0 2024-02-10T06:15:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.120+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2102756438 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc404bed0 con 0x7f9cc41000e0 2024-02-10T06:15:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.121+0000 7f9cc17fa700 1 -- 172.21.15.89:0/2102756438 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9cb0007cc0 con 0x7f9cc41000e0 2024-02-10T06:15:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.121+0000 7f9cc17fa700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9cb40692a0 0x7f9cb406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.121+0000 7f9cc17fa700 1 -- 172.21.15.89:0/2102756438 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9cb0086de0 con 0x7f9cc41000e0 2024-02-10T06:15:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.121+0000 7f9cc8831700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9cb40692a0 0x7f9cb406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.121+0000 7f9cc8831700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9cb40692a0 0x7f9cb406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800b0d0 tx=0x7f9cb8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.129+0000 7f9cc17fa700 1 -- 172.21.15.89:0/2102756438 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9cb0056a70 con 0x7f9cc41000e0 2024-02-10T06:15:47.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.357+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2102756438 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cc410f950 con 0x7f9cb40692a0 2024-02-10T06:15:47.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.359+0000 7f9cc17fa700 1 -- 172.21.15.89:0/2102756438 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f9cc410f950 con 0x7f9cb40692a0 2024-02-10T06:15:47.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.361+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2102756438 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9cb40692a0 msgr2=0x7f9cb406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:47.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.361+0000 7f9ccaa95700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9cb40692a0 0x7f9cb406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800b0d0 tx=0x7f9cb8009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:47.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.361+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2102756438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cc41000e0 msgr2=0x7f9cc410de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:47.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.361+0000 7f9ccaa95700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cc41000e0 0x7f9cc410de20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4066100 tx=0x7f9cb000b040 comp rx=0 tx=0).stop 2024-02-10T06:15:47.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.361+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2102756438 shutdown_connections 2024-02-10T06:15:47.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.361+0000 7f9ccaa95700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9cb40692a0 0x7f9cb406b750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.361+0000 7f9ccaa95700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc40663c0 0x7f9cc410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.361+0000 7f9ccaa95700 1 --2- 172.21.15.89:0/2102756438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cc41000e0 0x7f9cc410de20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.361+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2102756438 >> 172.21.15.89:0/2102756438 conn(0x7f9cc40f74e0 msgr2=0x7f9cc40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:47.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.361+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2102756438 shutdown_connections 2024-02-10T06:15:47.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.361+0000 7f9ccaa95700 1 -- 172.21.15.89:0/2102756438 wait complete. 2024-02-10T06:15:47.373 INFO:teuthology.orchestra.run.smithi089.stdout:true 2024-02-10T06:15:47.373 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:47 smithi089 ceph-mon[29217]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2024-02-10T06:15:47.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.470+0000 7fdb98c0b700 1 -- 172.21.15.89:0/1360186232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb940fbe40 msgr2=0x7fdb940fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:47.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.470+0000 7fdb98c0b700 1 --2- 172.21.15.89:0/1360186232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb940fbe40 0x7fdb940fc250 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdb84004940 tx=0x7fdb84009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:47.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.470+0000 7fdb98c0b700 1 -- 172.21.15.89:0/1360186232 shutdown_connections 2024-02-10T06:15:47.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.470+0000 7fdb98c0b700 1 --2- 172.21.15.89:0/1360186232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb940fcfb0 0x7fdb940fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.470+0000 7fdb98c0b700 1 --2- 172.21.15.89:0/1360186232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb940fbe40 0x7fdb940fc250 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.470+0000 7fdb98c0b700 1 -- 172.21.15.89:0/1360186232 >> 172.21.15.89:0/1360186232 conn(0x7fdb940f7500 msgr2=0x7fdb940f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.470+0000 7fdb98c0b700 1 -- 172.21.15.89:0/1360186232 shutdown_connections 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.470+0000 7fdb98c0b700 1 -- 172.21.15.89:0/1360186232 wait complete. 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb98c0b700 1 Processor -- start 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb98c0b700 1 -- start start 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb98c0b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb940fbe40 0x7fdb9410c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb98c0b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb940fcfb0 0x7fdb9410cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb98c0b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9410d310 con 0x7fdb940fcfb0 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb98c0b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9410dbb0 con 0x7fdb940fbe40 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb91d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb940fcfb0 0x7fdb9410cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb91d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb940fcfb0 0x7fdb9410cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37480/0 (socket says 172.21.15.89:37480) 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb91d9b700 1 -- 172.21.15.89:0/2786535288 learned_addr learned my addr 172.21.15.89:0/2786535288 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb9259c700 1 --2- 172.21.15.89:0/2786535288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb940fbe40 0x7fdb9410c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:47.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb91d9b700 1 -- 172.21.15.89:0/2786535288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb940fbe40 msgr2=0x7fdb9410c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:47.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb91d9b700 1 --2- 172.21.15.89:0/2786535288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb940fbe40 0x7fdb9410c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb91d9b700 1 -- 172.21.15.89:0/2786535288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb84005040 con 0x7fdb940fcfb0 2024-02-10T06:15:47.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.471+0000 7fdb91d9b700 1 --2- 172.21.15.89:0/2786535288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb940fcfb0 0x7fdb9410cd10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdb88010d50 tx=0x7fdb8800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:47.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.472+0000 7fdb82ffd700 1 -- 172.21.15.89:0/2786535288 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb8800ed60 con 0x7fdb940fcfb0 2024-02-10T06:15:47.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.472+0000 7fdb82ffd700 1 -- 172.21.15.89:0/2786535288 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb880090d0 con 0x7fdb940fcfb0 2024-02-10T06:15:47.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.472+0000 7fdb98c0b700 1 -- 172.21.15.89:0/2786535288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9410de30 con 0x7fdb940fcfb0 2024-02-10T06:15:47.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.472+0000 7fdb82ffd700 1 -- 172.21.15.89:0/2786535288 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb88012670 con 0x7fdb940fcfb0 2024-02-10T06:15:47.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.472+0000 7fdb98c0b700 1 -- 172.21.15.89:0/2786535288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9410e370 con 0x7fdb940fcfb0 2024-02-10T06:15:47.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.473+0000 7fdb98c0b700 1 -- 172.21.15.89:0/2786535288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9404bed0 con 0x7fdb940fcfb0 2024-02-10T06:15:47.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.473+0000 7fdb82ffd700 1 -- 172.21.15.89:0/2786535288 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fdb88018030 con 0x7fdb940fcfb0 2024-02-10T06:15:47.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.474+0000 7fdb82ffd700 1 --2- 172.21.15.89:0/2786535288 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdb7c0692a0 0x7fdb7c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:47.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.474+0000 7fdb82ffd700 1 -- 172.21.15.89:0/2786535288 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdb88016070 con 0x7fdb940fcfb0 2024-02-10T06:15:47.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.474+0000 7fdb9259c700 1 --2- 172.21.15.89:0/2786535288 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdb7c0692a0 0x7fdb7c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:47.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.474+0000 7fdb9259c700 1 --2- 172.21.15.89:0/2786535288 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdb7c0692a0 0x7fdb7c06b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009b30 tx=0x7fdb84009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:47.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.478+0000 7fdb82ffd700 1 -- 172.21.15.89:0/2786535288 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdb880595a0 con 0x7fdb940fcfb0 2024-02-10T06:15:47.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:47 smithi173 ceph-mon[36251]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2024-02-10T06:15:47.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.693+0000 7fdb98c0b700 1 -- 172.21.15.89:0/2786535288 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb9410e7c0 con 0x7fdb7c0692a0 2024-02-10T06:15:47.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.694+0000 7fdb82ffd700 1 -- 172.21.15.89:0/2786535288 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fdb9410e7c0 con 0x7fdb7c0692a0 2024-02-10T06:15:47.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.696+0000 7fdb98c0b700 1 -- 172.21.15.89:0/2786535288 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdb7c0692a0 msgr2=0x7fdb7c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:47.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.696+0000 7fdb98c0b700 1 --2- 172.21.15.89:0/2786535288 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdb7c0692a0 0x7fdb7c06b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009b30 tx=0x7fdb84009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:47.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.696+0000 7fdb98c0b700 1 -- 172.21.15.89:0/2786535288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb940fcfb0 msgr2=0x7fdb9410cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:47.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.696+0000 7fdb98c0b700 1 --2- 172.21.15.89:0/2786535288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb940fcfb0 0x7fdb9410cd10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdb88010d50 tx=0x7fdb8800b040 comp rx=0 tx=0).stop 2024-02-10T06:15:47.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.696+0000 7fdb98c0b700 1 -- 172.21.15.89:0/2786535288 shutdown_connections 2024-02-10T06:15:47.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.696+0000 7fdb98c0b700 1 --2- 172.21.15.89:0/2786535288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb940fbe40 0x7fdb9410c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.696+0000 7fdb98c0b700 1 --2- 172.21.15.89:0/2786535288 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fdb7c0692a0 0x7fdb7c06b750 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.696+0000 7fdb98c0b700 1 --2- 172.21.15.89:0/2786535288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb940fcfb0 0x7fdb9410cd10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.696+0000 7fdb98c0b700 1 -- 172.21.15.89:0/2786535288 >> 172.21.15.89:0/2786535288 conn(0x7fdb940f7500 msgr2=0x7fdb94100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:47.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.696+0000 7fdb98c0b700 1 -- 172.21.15.89:0/2786535288 shutdown_connections 2024-02-10T06:15:47.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.696+0000 7fdb98c0b700 1 -- 172.21.15.89:0/2786535288 wait complete. 2024-02-10T06:15:47.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.800+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/2422317358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d280fe0e0 msgr2=0x7f9d280fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:47.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.800+0000 7f9d2c8fe700 1 --2- 172.21.15.89:0/2422317358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d280fe0e0 0x7f9d280fe550 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c004940 tx=0x7f9d1c009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:47.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/2422317358 shutdown_connections 2024-02-10T06:15:47.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 --2- 172.21.15.89:0/2422317358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d280fe0e0 0x7f9d280fe550 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 --2- 172.21.15.89:0/2422317358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d280fcef0 0x7f9d280fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/2422317358 >> 172.21.15.89:0/2422317358 conn(0x7f9d280f8620 msgr2=0x7f9d280faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:47.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/2422317358 shutdown_connections 2024-02-10T06:15:47.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/2422317358 wait complete. 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 Processor -- start 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 -- start start 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d280fcef0 0x7f9d2810d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d280fe0e0 0x7f9d2810de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2810e410 con 0x7f9d280fcef0 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2c8fe700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2810ecb0 con 0x7f9d280fe0e0 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.801+0000 7f9d2659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d280fcef0 0x7f9d2810d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d2659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d280fcef0 0x7f9d2810d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37490/0 (socket says 172.21.15.89:37490) 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d2659c700 1 -- 172.21.15.89:0/3137418632 learned_addr learned my addr 172.21.15.89:0/3137418632 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d2659c700 1 -- 172.21.15.89:0/3137418632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d280fe0e0 msgr2=0x7f9d2810de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d25d9b700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d280fe0e0 0x7f9d2810de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d2659c700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d280fe0e0 0x7f9d2810de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d2659c700 1 -- 172.21.15.89:0/3137418632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c005040 con 0x7f9d280fcef0 2024-02-10T06:15:47.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d25d9b700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d280fe0e0 0x7f9d2810de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:15:47.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d2659c700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d280fcef0 0x7f9d2810d8d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800eb70 tx=0x7f9d1800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:47.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d16ffd700 1 -- 172.21.15.89:0/3137418632 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d18007cc0 con 0x7f9d280fcef0 2024-02-10T06:15:47.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d16ffd700 1 -- 172.21.15.89:0/3137418632 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d1800f040 con 0x7f9d280fcef0 2024-02-10T06:15:47.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/3137418632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2810ef30 con 0x7f9d280fcef0 2024-02-10T06:15:47.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d16ffd700 1 -- 172.21.15.89:0/3137418632 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d1800cbd0 con 0x7f9d280fcef0 2024-02-10T06:15:47.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.802+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/3137418632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2810f470 con 0x7f9d280fcef0 2024-02-10T06:15:47.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.803+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/3137418632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d2804bed0 con 0x7f9d280fcef0 2024-02-10T06:15:47.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.804+0000 7f9d16ffd700 1 -- 172.21.15.89:0/3137418632 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9d1800cd30 con 0x7f9d280fcef0 2024-02-10T06:15:47.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.804+0000 7f9d16ffd700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d100692b0 0x7f9d1006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:47.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.804+0000 7f9d16ffd700 1 -- 172.21.15.89:0/3137418632 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9d18086cc0 con 0x7f9d280fcef0 2024-02-10T06:15:47.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.804+0000 7f9d25d9b700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d100692b0 0x7f9d1006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:47.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.804+0000 7f9d25d9b700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d100692b0 0x7f9d1006b760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00b0d0 tx=0x7f9d1c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:47.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:47.808+0000 7f9d16ffd700 1 -- 172.21.15.89:0/3137418632 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9d1804dbd0 con 0x7f9d280fcef0 2024-02-10T06:15:48.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.006+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/3137418632 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d2810f0c0 con 0x7f9d100692b0 2024-02-10T06:15:48.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.015+0000 7f9d16ffd700 1 -- 172.21.15.89:0/3137418632 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f9d2810f0c0 con 0x7f9d100692b0 2024-02-10T06:15:48.016 INFO:teuthology.orchestra.run.smithi089.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:15:48.016 INFO:teuthology.orchestra.run.smithi089.stdout:alertmanager.smithi089 smithi089 *:9093,9094 running (5m) 62s ago 6m 24.6M - 0.25.0 c8568f914cd2 667ad25cb367 2024-02-10T06:15:48.016 INFO:teuthology.orchestra.run.smithi089.stdout:crash.smithi089 smithi089 running (6m) 62s ago 6m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 e3aac8b18020 2024-02-10T06:15:48.016 INFO:teuthology.orchestra.run.smithi089.stdout:crash.smithi173 smithi173 running (5m) 66s ago 5m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 269616f4b479 2024-02-10T06:15:48.016 INFO:teuthology.orchestra.run.smithi089.stdout:grafana.smithi089 smithi089 *:3000 running (5m) 62s ago 6m 86.1M - 9.4.7 954c08fa6188 a6f72269ee53 2024-02-10T06:15:48.016 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi089.evydlq smithi089 running (79s) 62s ago 78s 14.2M - 17.2.7-505-g72ce7c1d a2a0bd126c96 9e2f6f4766e6 2024-02-10T06:15:48.016 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi089.mgkvmj smithi089 running (86s) 62s ago 85s 10.4M - 17.2.7-505-g72ce7c1d a2a0bd126c96 689a4ca9f11b 2024-02-10T06:15:48.016 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi173.cjkmdf smithi173 running (76s) 66s ago 76s 12.0M - 17.2.7-505-g72ce7c1d a2a0bd126c96 3d9eb72f6f12 2024-02-10T06:15:48.016 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi173.tumnkz smithi173 running (83s) 66s ago 83s 17.1M - 17.2.7-505-g72ce7c1d a2a0bd126c96 272e93861538 2024-02-10T06:15:48.016 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.smithi089.pwmrye smithi089 *:9283 running (7m) 62s ago 7m 481M - 17.2.7-505-g72ce7c1d a2a0bd126c96 1cd3521b2ebb 2024-02-10T06:15:48.016 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.smithi173.swdirm smithi173 *:8443,9283 running (5m) 66s ago 5m 433M - 17.2.7-505-g72ce7c1d a2a0bd126c96 6430108d8d2e 2024-02-10T06:15:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:mon.smithi089 smithi089 running (8m) 62s ago 8m 46.7M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 0109d6119759 2024-02-10T06:15:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:mon.smithi173 smithi173 running (5m) 66s ago 5m 35.8M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 08af8aefe37b 2024-02-10T06:15:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:node-exporter.smithi089 smithi089 *:9100 running (6m) 62s ago 6m 14.7M - 1.5.0 0da6a335fe13 18b1f6a49022 2024-02-10T06:15:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:node-exporter.smithi173 smithi173 *:9100 running (5m) 66s ago 5m 16.5M - 1.5.0 0da6a335fe13 f7d0b906813f 2024-02-10T06:15:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0 smithi089 running (4m) 62s ago 4m 47.5M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 e372a21351db 2024-02-10T06:15:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1 smithi089 running (4m) 62s ago 4m 46.3M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 42f862e2f6ca 2024-02-10T06:15:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2 smithi089 running (4m) 62s ago 4m 45.2M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 69b16e6f89fb 2024-02-10T06:15:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3 smithi173 running (3m) 66s ago 3m 43.3M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 997c2081faec 2024-02-10T06:15:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4 smithi173 running (3m) 66s ago 3m 42.0M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 d3c6d0e77356 2024-02-10T06:15:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5 smithi173 running (2m) 66s ago 2m 41.2M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 bb845c14ca98 2024-02-10T06:15:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:prometheus.smithi089 smithi089 *:9095 running (5m) 62s ago 6m 53.8M - 2.43.0 a07b618ecd1d 04539fe88109 2024-02-10T06:15:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.017+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/3137418632 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d100692b0 msgr2=0x7f9d1006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.017+0000 7f9d2c8fe700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d100692b0 0x7f9d1006b760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00b0d0 tx=0x7f9d1c009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.017+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/3137418632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d280fcef0 msgr2=0x7f9d2810d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.017+0000 7f9d2c8fe700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d280fcef0 0x7f9d2810d8d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800eb70 tx=0x7f9d1800b040 comp rx=0 tx=0).stop 2024-02-10T06:15:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.017+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/3137418632 shutdown_connections 2024-02-10T06:15:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.017+0000 7f9d2c8fe700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f9d100692b0 0x7f9d1006b760 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.017+0000 7f9d2c8fe700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d280fcef0 0x7f9d2810d8d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.017+0000 7f9d2c8fe700 1 --2- 172.21.15.89:0/3137418632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d280fe0e0 0x7f9d2810de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.017+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/3137418632 >> 172.21.15.89:0/3137418632 conn(0x7f9d280f8620 msgr2=0x7f9d281013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.017+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/3137418632 shutdown_connections 2024-02-10T06:15:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.017+0000 7f9d2c8fe700 1 -- 172.21.15.89:0/3137418632 wait complete. 2024-02-10T06:15:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.116+0000 7f92c9223700 1 -- 172.21.15.89:0/938065044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92c40fbe40 msgr2=0x7f92c40fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.116+0000 7f92c9223700 1 --2- 172.21.15.89:0/938065044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92c40fbe40 0x7f92c40fc250 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009ce0 tx=0x7f92bc004d10 comp rx=0 tx=0).stop 2024-02-10T06:15:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.117+0000 7f92c9223700 1 -- 172.21.15.89:0/938065044 shutdown_connections 2024-02-10T06:15:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.117+0000 7f92c9223700 1 --2- 172.21.15.89:0/938065044 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92c40fcfb0 0x7f92c40fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.117+0000 7f92c9223700 1 --2- 172.21.15.89:0/938065044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92c40fbe40 0x7f92c40fc250 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.117+0000 7f92c9223700 1 -- 172.21.15.89:0/938065044 >> 172.21.15.89:0/938065044 conn(0x7f92c40f74e0 msgr2=0x7f92c40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.117+0000 7f92c9223700 1 -- 172.21.15.89:0/938065044 shutdown_connections 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.117+0000 7f92c9223700 1 -- 172.21.15.89:0/938065044 wait complete. 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.117+0000 7f92c9223700 1 Processor -- start 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.117+0000 7f92c9223700 1 -- start start 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.117+0000 7f92c9223700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92c40fbe40 0x7f92c410db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.117+0000 7f92c9223700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92c40fcfb0 0x7f92c410e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c9223700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c410e650 con 0x7f92c40fbe40 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c9223700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c410eef0 con 0x7f92c40fcfb0 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92c40fbe40 0x7f92c410db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c2d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92c40fbe40 0x7f92c410db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37506/0 (socket says 172.21.15.89:37506) 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c2d9d700 1 -- 172.21.15.89:0/3601614827 learned_addr learned my addr 172.21.15.89:0/3601614827 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c259c700 1 --2- 172.21.15.89:0/3601614827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92c40fcfb0 0x7f92c410e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c2d9d700 1 -- 172.21.15.89:0/3601614827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92c40fcfb0 msgr2=0x7f92c410e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c2d9d700 1 --2- 172.21.15.89:0/3601614827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92c40fcfb0 0x7f92c410e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c2d9d700 1 -- 172.21.15.89:0/3601614827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc009940 con 0x7f92c40fbe40 2024-02-10T06:15:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c2d9d700 1 --2- 172.21.15.89:0/3601614827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92c40fbe40 0x7f92c410db10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f92bc015040 tx=0x7f92bc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92bb7fe700 1 -- 172.21.15.89:0/3601614827 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92bc00e040 con 0x7f92c40fbe40 2024-02-10T06:15:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c9223700 1 -- 172.21.15.89:0/3601614827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c410f170 con 0x7f92c40fbe40 2024-02-10T06:15:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92bb7fe700 1 -- 172.21.15.89:0/3601614827 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92bc008e20 con 0x7f92c40fbe40 2024-02-10T06:15:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92bb7fe700 1 -- 172.21.15.89:0/3601614827 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92bc01c710 con 0x7f92c40fbe40 2024-02-10T06:15:48.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.118+0000 7f92c9223700 1 -- 172.21.15.89:0/3601614827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c410f6b0 con 0x7f92c40fbe40 2024-02-10T06:15:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.119+0000 7f92c9223700 1 -- 172.21.15.89:0/3601614827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c404bed0 con 0x7f92c40fbe40 2024-02-10T06:15:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.120+0000 7f92bb7fe700 1 -- 172.21.15.89:0/3601614827 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f92bc012b00 con 0x7f92c40fbe40 2024-02-10T06:15:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.120+0000 7f92bb7fe700 1 --2- 172.21.15.89:0/3601614827 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92b00692b0 0x7f92b006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.120+0000 7f92bb7fe700 1 -- 172.21.15.89:0/3601614827 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f92bc088db0 con 0x7f92c40fbe40 2024-02-10T06:15:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.120+0000 7f92c259c700 1 --2- 172.21.15.89:0/3601614827 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92b00692b0 0x7f92b006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:48.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.120+0000 7f92c259c700 1 --2- 172.21.15.89:0/3601614827 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92b00692b0 0x7f92b006b760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f92ac004f40 tx=0x7f92ac00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:48.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.124+0000 7f92bb7fe700 1 -- 172.21.15.89:0/3601614827 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f92bc058aa0 con 0x7f92c40fbe40 2024-02-10T06:15:48.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:48 smithi089 ceph-mon[29217]: from='client.24321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:48.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.393+0000 7f92c9223700 1 -- 172.21.15.89:0/3601614827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f92c410f300 con 0x7f92c40fbe40 2024-02-10T06:15:48.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.393+0000 7f92bb7fe700 1 -- 172.21.15.89:0/3601614827 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f92c410f300 con 0x7f92c40fbe40 2024-02-10T06:15:48.394 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-10T06:15:48.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": { 2024-02-10T06:15:48.394 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-10T06:15:48.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:15:48.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": { 2024-02-10T06:15:48.394 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-10T06:15:48.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:15:48.394 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": { 2024-02-10T06:15:48.395 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 6 2024-02-10T06:15:48.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:15:48.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": { 2024-02-10T06:15:48.395 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 4 2024-02-10T06:15:48.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:15:48.395 INFO:teuthology.orchestra.run.smithi089.stdout: "overall": { 2024-02-10T06:15:48.395 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 14 2024-02-10T06:15:48.395 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-10T06:15:48.395 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-10T06:15:48.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.396+0000 7f92c9223700 1 -- 172.21.15.89:0/3601614827 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92b00692b0 msgr2=0x7f92b006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.396+0000 7f92c9223700 1 --2- 172.21.15.89:0/3601614827 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92b00692b0 0x7f92b006b760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f92ac004f40 tx=0x7f92ac00b040 comp rx=0 tx=0).stop 2024-02-10T06:15:48.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.396+0000 7f92c9223700 1 -- 172.21.15.89:0/3601614827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92c40fbe40 msgr2=0x7f92c410db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.396+0000 7f92c9223700 1 --2- 172.21.15.89:0/3601614827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92c40fbe40 0x7f92c410db10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f92bc015040 tx=0x7f92bc004d10 comp rx=0 tx=0).stop 2024-02-10T06:15:48.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.396+0000 7f92c9223700 1 -- 172.21.15.89:0/3601614827 shutdown_connections 2024-02-10T06:15:48.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.396+0000 7f92c9223700 1 --2- 172.21.15.89:0/3601614827 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f92b00692b0 0x7f92b006b760 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.396+0000 7f92c9223700 1 --2- 172.21.15.89:0/3601614827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92c40fbe40 0x7f92c410db10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.396+0000 7f92c9223700 1 --2- 172.21.15.89:0/3601614827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92c40fcfb0 0x7f92c410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.396+0000 7f92c9223700 1 -- 172.21.15.89:0/3601614827 >> 172.21.15.89:0/3601614827 conn(0x7f92c40f74e0 msgr2=0x7f92c40656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:48.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.397+0000 7f92c9223700 1 -- 172.21.15.89:0/3601614827 shutdown_connections 2024-02-10T06:15:48.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.397+0000 7f92c9223700 1 -- 172.21.15.89:0/3601614827 wait complete. 2024-02-10T06:15:48.399 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:48 smithi173 ceph-mon[36251]: from='client.24321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:48.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.507+0000 7f18caa0f700 1 -- 172.21.15.89:0/1393256315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18c40fcfb0 msgr2=0x7f18c40fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.507+0000 7f18caa0f700 1 --2- 172.21.15.89:0/1393256315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18c40fcfb0 0x7f18c40fd420 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f18b8004940 tx=0x7f18b8009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:48.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.507+0000 7f18caa0f700 1 -- 172.21.15.89:0/1393256315 shutdown_connections 2024-02-10T06:15:48.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.507+0000 7f18caa0f700 1 --2- 172.21.15.89:0/1393256315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18c40fcfb0 0x7f18c40fd420 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.507+0000 7f18caa0f700 1 --2- 172.21.15.89:0/1393256315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c40fbe40 0x7f18c40fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.507+0000 7f18caa0f700 1 -- 172.21.15.89:0/1393256315 >> 172.21.15.89:0/1393256315 conn(0x7f18c40f7520 msgr2=0x7f18c40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:48.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.507+0000 7f18caa0f700 1 -- 172.21.15.89:0/1393256315 shutdown_connections 2024-02-10T06:15:48.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.507+0000 7f18caa0f700 1 -- 172.21.15.89:0/1393256315 wait complete. 2024-02-10T06:15:48.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.507+0000 7f18caa0f700 1 Processor -- start 2024-02-10T06:15:48.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18caa0f700 1 -- start start 2024-02-10T06:15:48.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18caa0f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18c40fbe40 0x7f18c41105c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:48.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18caa0f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c40fcfb0 0x7f18c4112b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:48.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18caa0f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4066cf0 con 0x7f18c40fbe40 2024-02-10T06:15:48.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18caa0f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4066e30 con 0x7f18c40fcfb0 2024-02-10T06:15:48.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18c3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18c40fbe40 0x7f18c41105c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:48.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18c3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18c40fbe40 0x7f18c41105c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49362/0 (socket says 172.21.15.89:49362) 2024-02-10T06:15:48.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18c3fff700 1 -- 172.21.15.89:0/2185736986 learned_addr learned my addr 172.21.15.89:0/2185736986 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:48.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18c37fe700 1 --2- 172.21.15.89:0/2185736986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c40fcfb0 0x7f18c4112b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:48.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18c3fff700 1 -- 172.21.15.89:0/2185736986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c40fcfb0 msgr2=0x7f18c4112b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18c3fff700 1 --2- 172.21.15.89:0/2185736986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c40fcfb0 0x7f18c4112b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.508+0000 7f18c3fff700 1 -- 172.21.15.89:0/2185736986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b8005040 con 0x7f18c40fbe40 2024-02-10T06:15:48.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.509+0000 7f18c3fff700 1 --2- 172.21.15.89:0/2185736986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18c40fbe40 0x7f18c41105c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f18b400eb40 tx=0x7f18b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:48.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.509+0000 7f18c0ff9700 1 -- 172.21.15.89:0/2185736986 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18b4007cc0 con 0x7f18c40fbe40 2024-02-10T06:15:48.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.509+0000 7f18c0ff9700 1 -- 172.21.15.89:0/2185736986 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18b400f040 con 0x7f18c40fbe40 2024-02-10T06:15:48.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.509+0000 7f18caa0f700 1 -- 172.21.15.89:0/2185736986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c4113110 con 0x7f18c40fbe40 2024-02-10T06:15:48.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.509+0000 7f18c0ff9700 1 -- 172.21.15.89:0/2185736986 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18b400ca10 con 0x7f18c40fbe40 2024-02-10T06:15:48.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.509+0000 7f18caa0f700 1 -- 172.21.15.89:0/2185736986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c4113650 con 0x7f18c40fbe40 2024-02-10T06:15:48.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.510+0000 7f18caa0f700 1 -- 172.21.15.89:0/2185736986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c404bed0 con 0x7f18c40fbe40 2024-02-10T06:15:48.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.511+0000 7f18c0ff9700 1 -- 172.21.15.89:0/2185736986 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f18b400cc80 con 0x7f18c40fbe40 2024-02-10T06:15:48.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.511+0000 7f18c0ff9700 1 --2- 172.21.15.89:0/2185736986 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18ac0692a0 0x7f18ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:48.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.511+0000 7f18c0ff9700 1 -- 172.21.15.89:0/2185736986 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f18b4086dd0 con 0x7f18c40fbe40 2024-02-10T06:15:48.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.511+0000 7f18c37fe700 1 --2- 172.21.15.89:0/2185736986 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18ac0692a0 0x7f18ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:48.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.512+0000 7f18c37fe700 1 --2- 172.21.15.89:0/2185736986 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18ac0692a0 0x7f18ac06b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009b30 tx=0x7f18b8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:48.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.515+0000 7f18c0ff9700 1 -- 172.21.15.89:0/2185736986 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f18b404dbd0 con 0x7f18c40fbe40 2024-02-10T06:15:48.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.752+0000 7f18caa0f700 1 -- 172.21.15.89:0/2185736986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f18c4113a90 con 0x7f18c40fbe40 2024-02-10T06:15:48.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.752+0000 7f18c0ff9700 1 -- 172.21.15.89:0/2185736986 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1838 (secure 0 0 0) 0x7f18b40530e0 con 0x7f18c40fbe40 2024-02-10T06:15:48.753 INFO:teuthology.orchestra.run.smithi089.stdout:e15 2024-02-10T06:15:48.753 INFO:teuthology.orchestra.run.smithi089.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:15:48.753 INFO:teuthology.orchestra.run.smithi089.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:15:48.753 INFO:teuthology.orchestra.run.smithi089.stdout:legacy client fscid: 1 2024-02-10T06:15:48.753 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:15:48.753 INFO:teuthology.orchestra.run.smithi089.stdout:Filesystem 'cephfs' (1) 2024-02-10T06:15:48.753 INFO:teuthology.orchestra.run.smithi089.stdout:fs_name cephfs 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 15 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-02-10T06:14:18.236041+0000 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:modified 2024-02-10T06:14:36.018945+0000 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:tableserver 0 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:root 0 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:session_timeout 60 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:session_autoclose 300 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:max_file_size 1099511627776 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:required_client_features {} 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure 0 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure_osd_epoch 47 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:max_mds 1 2024-02-10T06:15:48.754 INFO:teuthology.orchestra.run.smithi089.stdout:in 0 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:up {0=24235} 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:failed 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:damaged 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:stopped 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:data_pools [3] 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:metadata_pool 2 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:inline_data enabled 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:balancer 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:standby_count_wanted 1 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi173.tumnkz{0:24235} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi089.evydlq{0:14476} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.89:6828/3386828748,v1:172.21.15.89:6829/3386828748] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout:Standby daemons: 2024-02-10T06:15:48.755 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:15:48.756 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi173.cjkmdf{-1:24243} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.173:6826/2798078299,v1:172.21.15.173:6827/2798078299] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:15:48.756 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi089.mgkvmj{-1:24251} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.89:6826/3574254294,v1:172.21.15.89:6827/3574254294] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:15:48.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.754+0000 7f18caa0f700 1 -- 172.21.15.89:0/2185736986 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18ac0692a0 msgr2=0x7f18ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.754+0000 7f18caa0f700 1 --2- 172.21.15.89:0/2185736986 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18ac0692a0 0x7f18ac06b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009b30 tx=0x7f18b8009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.754+0000 7f18caa0f700 1 -- 172.21.15.89:0/2185736986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18c40fbe40 msgr2=0x7f18c41105c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.754+0000 7f18caa0f700 1 --2- 172.21.15.89:0/2185736986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18c40fbe40 0x7f18c41105c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f18b400eb40 tx=0x7f18b400b040 comp rx=0 tx=0).stop 2024-02-10T06:15:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.754+0000 7f18caa0f700 1 -- 172.21.15.89:0/2185736986 shutdown_connections 2024-02-10T06:15:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.754+0000 7f18caa0f700 1 --2- 172.21.15.89:0/2185736986 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f18ac0692a0 0x7f18ac06b750 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.754+0000 7f18caa0f700 1 --2- 172.21.15.89:0/2185736986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18c40fbe40 0x7f18c41105c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.754+0000 7f18caa0f700 1 --2- 172.21.15.89:0/2185736986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c40fcfb0 0x7f18c4112b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.754+0000 7f18caa0f700 1 -- 172.21.15.89:0/2185736986 >> 172.21.15.89:0/2185736986 conn(0x7f18c40f7520 msgr2=0x7f18c4100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.755+0000 7f18caa0f700 1 -- 172.21.15.89:0/2185736986 shutdown_connections 2024-02-10T06:15:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.755+0000 7f18caa0f700 1 -- 172.21.15.89:0/2185736986 wait complete. 2024-02-10T06:15:48.757 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 15 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.858+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/1485311924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d840fcfc0 msgr2=0x7f0d840fd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.858+0000 7f0d8b1ea700 1 --2- 172.21.15.89:0/1485311924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d840fcfc0 0x7f0d840fd430 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0d70004940 tx=0x7f0d70009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.858+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/1485311924 shutdown_connections 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.858+0000 7f0d8b1ea700 1 --2- 172.21.15.89:0/1485311924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d840fcfc0 0x7f0d840fd430 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.858+0000 7f0d8b1ea700 1 --2- 172.21.15.89:0/1485311924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d840fbe50 0x7f0d840fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.858+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/1485311924 >> 172.21.15.89:0/1485311924 conn(0x7f0d840f7510 msgr2=0x7f0d840f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.858+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/1485311924 shutdown_connections 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.858+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/1485311924 wait complete. 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.859+0000 7f0d8b1ea700 1 Processor -- start 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.859+0000 7f0d8b1ea700 1 -- start start 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.859+0000 7f0d8b1ea700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d840fbe50 0x7f0d841129d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:48.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.859+0000 7f0d8b1ea700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d840fcfc0 0x7f0d84114f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.859+0000 7f0d8b1ea700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d84066be0 con 0x7f0d840fcfc0 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.859+0000 7f0d8b1ea700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d84066d20 con 0x7f0d840fbe50 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.859+0000 7f0d83fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d840fcfc0 0x7f0d84114f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.859+0000 7f0d88f86700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d840fbe50 0x7f0d841129d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.859+0000 7f0d83fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d840fcfc0 0x7f0d84114f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49374/0 (socket says 172.21.15.89:49374) 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.859+0000 7f0d83fff700 1 -- 172.21.15.89:0/2825380232 learned_addr learned my addr 172.21.15.89:0/2825380232 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.860+0000 7f0d83fff700 1 -- 172.21.15.89:0/2825380232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d840fbe50 msgr2=0x7f0d841129d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.860+0000 7f0d83fff700 1 --2- 172.21.15.89:0/2825380232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d840fbe50 0x7f0d841129d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.860+0000 7f0d83fff700 1 -- 172.21.15.89:0/2825380232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d70005040 con 0x7f0d840fcfc0 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.860+0000 7f0d88f86700 1 --2- 172.21.15.89:0/2825380232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d840fbe50 0x7f0d841129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.860+0000 7f0d83fff700 1 --2- 172.21.15.89:0/2825380232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d840fcfc0 0x7f0d84114f20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0d70000c00 tx=0x7f0d70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.860+0000 7f0d817fa700 1 -- 172.21.15.89:0/2825380232 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d70021070 con 0x7f0d840fcfc0 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.860+0000 7f0d817fa700 1 -- 172.21.15.89:0/2825380232 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d7001c400 con 0x7f0d840fcfc0 2024-02-10T06:15:48.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.860+0000 7f0d817fa700 1 -- 172.21.15.89:0/2825380232 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7000e070 con 0x7f0d840fcfc0 2024-02-10T06:15:48.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.860+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/2825380232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d84115460 con 0x7f0d840fcfc0 2024-02-10T06:15:48.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.860+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/2825380232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d84115800 con 0x7f0d840fcfc0 2024-02-10T06:15:48.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.861+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/2825380232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8404bed0 con 0x7f0d840fcfc0 2024-02-10T06:15:48.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.862+0000 7f0d817fa700 1 -- 172.21.15.89:0/2825380232 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0d7001ca50 con 0x7f0d840fcfc0 2024-02-10T06:15:48.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.862+0000 7f0d817fa700 1 --2- 172.21.15.89:0/2825380232 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0d740692a0 0x7f0d7406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:48.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.863+0000 7f0d817fa700 1 -- 172.21.15.89:0/2825380232 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0d70088de0 con 0x7f0d840fcfc0 2024-02-10T06:15:48.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.863+0000 7f0d88f86700 1 --2- 172.21.15.89:0/2825380232 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0d740692a0 0x7f0d7406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:48.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.863+0000 7f0d88f86700 1 --2- 172.21.15.89:0/2825380232 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0d740692a0 0x7f0d7406b750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0d84066970 tx=0x7f0d7800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:48.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:48.866+0000 7f0d817fa700 1 -- 172.21.15.89:0/2825380232 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0d70055160 con 0x7f0d840fcfc0 2024-02-10T06:15:49.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.071+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/2825380232 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d841019a0 con 0x7f0d740692a0 2024-02-10T06:15:49.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.072+0000 7f0d817fa700 1 -- 172.21.15.89:0/2825380232 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f0d841019a0 con 0x7f0d740692a0 2024-02-10T06:15:49.073 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-10T06:15:49.073 INFO:teuthology.orchestra.run.smithi089.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:15:49.073 INFO:teuthology.orchestra.run.smithi089.stdout: "in_progress": true, 2024-02-10T06:15:49.073 INFO:teuthology.orchestra.run.smithi089.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:15:49.073 INFO:teuthology.orchestra.run.smithi089.stdout: "services_complete": [], 2024-02-10T06:15:49.073 INFO:teuthology.orchestra.run.smithi089.stdout: "progress": "0/21 daemons upgraded", 2024-02-10T06:15:49.073 INFO:teuthology.orchestra.run.smithi089.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 image on host smithi173", 2024-02-10T06:15:49.073 INFO:teuthology.orchestra.run.smithi089.stdout: "is_paused": false 2024-02-10T06:15:49.074 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-10T06:15:49.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.074+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/2825380232 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0d740692a0 msgr2=0x7f0d7406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:49.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.074+0000 7f0d8b1ea700 1 --2- 172.21.15.89:0/2825380232 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0d740692a0 0x7f0d7406b750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0d84066970 tx=0x7f0d7800b040 comp rx=0 tx=0).stop 2024-02-10T06:15:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.074+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/2825380232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d840fcfc0 msgr2=0x7f0d84114f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.074+0000 7f0d8b1ea700 1 --2- 172.21.15.89:0/2825380232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d840fcfc0 0x7f0d84114f20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0d70000c00 tx=0x7f0d70009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.075+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/2825380232 shutdown_connections 2024-02-10T06:15:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.075+0000 7f0d8b1ea700 1 --2- 172.21.15.89:0/2825380232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d840fbe50 0x7f0d841129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.075+0000 7f0d8b1ea700 1 --2- 172.21.15.89:0/2825380232 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0d740692a0 0x7f0d7406b750 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.075+0000 7f0d8b1ea700 1 --2- 172.21.15.89:0/2825380232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d840fcfc0 0x7f0d84114f20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.075+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/2825380232 >> 172.21.15.89:0/2825380232 conn(0x7f0d840f7510 msgr2=0x7f0d84100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.075+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/2825380232 shutdown_connections 2024-02-10T06:15:49.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.075+0000 7f0d8b1ea700 1 -- 172.21.15.89:0/2825380232 wait complete. 2024-02-10T06:15:49.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.179+0000 7f822f35d700 1 -- 172.21.15.89:0/325633522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82280fcfb0 msgr2=0x7f82280fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.179+0000 7f822f35d700 1 --2- 172.21.15.89:0/325633522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82280fcfb0 0x7f82280fd420 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8224004940 tx=0x7f8224009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.179+0000 7f822f35d700 1 -- 172.21.15.89:0/325633522 shutdown_connections 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.179+0000 7f822f35d700 1 --2- 172.21.15.89:0/325633522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82280fcfb0 0x7f82280fd420 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.179+0000 7f822f35d700 1 --2- 172.21.15.89:0/325633522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82280fbe40 0x7f82280fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.179+0000 7f822f35d700 1 -- 172.21.15.89:0/325633522 >> 172.21.15.89:0/325633522 conn(0x7f82280f74e0 msgr2=0x7f82280f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.179+0000 7f822f35d700 1 -- 172.21.15.89:0/325633522 shutdown_connections 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.179+0000 7f822f35d700 1 -- 172.21.15.89:0/325633522 wait complete. 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822f35d700 1 Processor -- start 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822f35d700 1 -- start start 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822f35d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82280fbe40 0x7f82281105d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822f35d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82280fcfb0 0x7f8228112b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:49.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822f35d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8228066480 con 0x7f82280fcfb0 2024-02-10T06:15:49.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822f35d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82280665c0 con 0x7f82280fbe40 2024-02-10T06:15:49.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822c8f8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82280fcfb0 0x7f8228112b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:49.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822c8f8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82280fcfb0 0x7f8228112b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49386/0 (socket says 172.21.15.89:49386) 2024-02-10T06:15:49.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822c8f8700 1 -- 172.21.15.89:0/1042483598 learned_addr learned my addr 172.21.15.89:0/1042483598 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:15:49.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822d0f9700 1 --2- 172.21.15.89:0/1042483598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82280fbe40 0x7f82281105d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:49.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822c8f8700 1 -- 172.21.15.89:0/1042483598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82280fbe40 msgr2=0x7f82281105d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:49.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822c8f8700 1 --2- 172.21.15.89:0/1042483598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82280fbe40 0x7f82281105d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:49.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.180+0000 7f822c8f8700 1 -- 172.21.15.89:0/1042483598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8224005040 con 0x7f82280fcfb0 2024-02-10T06:15:49.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.181+0000 7f822c8f8700 1 --2- 172.21.15.89:0/1042483598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82280fcfb0 0x7f8228112b20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8224000c00 tx=0x7f8224009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:49.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.181+0000 7f821dffb700 1 -- 172.21.15.89:0/1042483598 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8224021070 con 0x7f82280fcfb0 2024-02-10T06:15:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.181+0000 7f821dffb700 1 -- 172.21.15.89:0/1042483598 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f822401c400 con 0x7f82280fcfb0 2024-02-10T06:15:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.181+0000 7f821dffb700 1 -- 172.21.15.89:0/1042483598 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f822400e070 con 0x7f82280fcfb0 2024-02-10T06:15:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.181+0000 7f822f35d700 1 -- 172.21.15.89:0/1042483598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8228113120 con 0x7f82280fcfb0 2024-02-10T06:15:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.181+0000 7f822f35d700 1 -- 172.21.15.89:0/1042483598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8228113580 con 0x7f82280fcfb0 2024-02-10T06:15:49.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.183+0000 7f822f35d700 1 -- 172.21.15.89:0/1042483598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822804bed0 con 0x7f82280fcfb0 2024-02-10T06:15:49.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.183+0000 7f821dffb700 1 -- 172.21.15.89:0/1042483598 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f822401ca50 con 0x7f82280fcfb0 2024-02-10T06:15:49.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.183+0000 7f821dffb700 1 --2- 172.21.15.89:0/1042483598 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f82140692a0 0x7f821406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:15:49.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.184+0000 7f821dffb700 1 -- 172.21.15.89:0/1042483598 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8224088480 con 0x7f82280fcfb0 2024-02-10T06:15:49.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.184+0000 7f822d0f9700 1 --2- 172.21.15.89:0/1042483598 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f82140692a0 0x7f821406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:15:49.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.184+0000 7f822d0f9700 1 --2- 172.21.15.89:0/1042483598 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f82140692a0 0x7f821406b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8228066100 tx=0x7f821800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:15:49.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.188+0000 7f821dffb700 1 -- 172.21.15.89:0/1042483598 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f82240546c0 con 0x7f82280fcfb0 2024-02-10T06:15:49.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:49 smithi089 ceph-mon[29217]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:49.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:49 smithi089 ceph-mon[29217]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:49.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:49 smithi089 ceph-mon[29217]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-10T06:15:49.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:49 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/3601614827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:15:49.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:49 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/2185736986' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:15:49.371 INFO:tasks.workunit.client.0.smithi089.stderr:Updating files: 98% (11971/12104) Updating files: 99% (11983/12104) Updating files: 100% (12104/12104) Updating files: 100% (12104/12104), done. 2024-02-10T06:15:49.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.469+0000 7f822f35d700 1 -- 172.21.15.89:0/1042483598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8228113a40 con 0x7f82280fcfb0 2024-02-10T06:15:49.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.469+0000 7f821dffb700 1 -- 172.21.15.89:0/1042483598 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f8224057ce0 con 0x7f82280fcfb0 2024-02-10T06:15:49.470 INFO:teuthology.orchestra.run.smithi089.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-10T06:15:49.470 INFO:teuthology.orchestra.run.smithi089.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-10T06:15:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-10T06:15:49.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.471+0000 7f822f35d700 1 -- 172.21.15.89:0/1042483598 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f82140692a0 msgr2=0x7f821406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:49.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.471+0000 7f822f35d700 1 --2- 172.21.15.89:0/1042483598 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f82140692a0 0x7f821406b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8228066100 tx=0x7f821800b040 comp rx=0 tx=0).stop 2024-02-10T06:15:49.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.471+0000 7f822f35d700 1 -- 172.21.15.89:0/1042483598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82280fcfb0 msgr2=0x7f8228112b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:15:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.471+0000 7f822f35d700 1 --2- 172.21.15.89:0/1042483598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82280fcfb0 0x7f8228112b20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8224000c00 tx=0x7f8224009e30 comp rx=0 tx=0).stop 2024-02-10T06:15:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.471+0000 7f822f35d700 1 -- 172.21.15.89:0/1042483598 shutdown_connections 2024-02-10T06:15:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.471+0000 7f822f35d700 1 --2- 172.21.15.89:0/1042483598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82280fbe40 0x7f82281105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.471+0000 7f822f35d700 1 --2- 172.21.15.89:0/1042483598 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f82140692a0 0x7f821406b750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.471+0000 7f822f35d700 1 --2- 172.21.15.89:0/1042483598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82280fcfb0 0x7f8228112b20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:15:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.471+0000 7f822f35d700 1 -- 172.21.15.89:0/1042483598 >> 172.21.15.89:0/1042483598 conn(0x7f82280f74e0 msgr2=0x7f8228100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:15:49.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.471+0000 7f822f35d700 1 -- 172.21.15.89:0/1042483598 shutdown_connections 2024-02-10T06:15:49.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:15:49.471+0000 7f822f35d700 1 -- 172.21.15.89:0/1042483598 wait complete. 2024-02-10T06:15:49.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:49 smithi173 ceph-mon[36251]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:49.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:49 smithi173 ceph-mon[36251]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:49.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:49 smithi173 ceph-mon[36251]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-10T06:15:49.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:49 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/3601614827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T06:15:49.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:49 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/2185736986' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-10T06:15:50.057 INFO:tasks.workunit.client.0.smithi089.stderr:Note: switching to '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'. 2024-02-10T06:15:50.057 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-02-10T06:15:50.057 INFO:tasks.workunit.client.0.smithi089.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-10T06:15:50.057 INFO:tasks.workunit.client.0.smithi089.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-10T06:15:50.057 INFO:tasks.workunit.client.0.smithi089.stderr:state without impacting any branches by switching back to a branch. 2024-02-10T06:15:50.057 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-02-10T06:15:50.057 INFO:tasks.workunit.client.0.smithi089.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-10T06:15:50.057 INFO:tasks.workunit.client.0.smithi089.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-10T06:15:50.057 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-02-10T06:15:50.057 INFO:tasks.workunit.client.0.smithi089.stderr: git switch -c 2024-02-10T06:15:50.057 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-02-10T06:15:50.058 INFO:tasks.workunit.client.0.smithi089.stderr:Or undo this operation with: 2024-02-10T06:15:50.058 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-02-10T06:15:50.058 INFO:tasks.workunit.client.0.smithi089.stderr: git switch - 2024-02-10T06:15:50.058 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-02-10T06:15:50.058 INFO:tasks.workunit.client.0.smithi089.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-10T06:15:50.058 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-02-10T06:15:50.058 INFO:tasks.workunit.client.0.smithi089.stderr:HEAD is now at 290212a1bfd Merge pull request #53521 from xxhdx1985126/wip-62857 2024-02-10T06:15:50.061 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-10T06:15:50.086 INFO:tasks.workunit.client.0.smithi089.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-10T06:15:50.088 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-10T06:15:50.088 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-10T06:15:50.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:50 smithi089 ceph-mon[29217]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:50.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:50 smithi089 ceph-mon[29217]: from='client.? 172.21.15.89:0/1042483598' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:15:50.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:50 smithi173 ceph-mon[36251]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T06:15:50.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:50 smithi173 ceph-mon[36251]: from='client.? 172.21.15.89:0/1042483598' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T06:15:50.892 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-10T06:15:50.994 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-10T06:15:51.066 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-10T06:15:51.068 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-10T06:15:51.068 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-10T06:15:51.220 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-10T06:15:51.224 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:15:51.224 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-10T06:15:51.289 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-02-10T06:15:51.290 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-10T06:15:51.291 DEBUG:teuthology.orchestra.run.smithi089:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-02-10T06:15:51.359 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir -p fsstress 2024-02-10T06:15:51.363 INFO:tasks.workunit.client.0.smithi089.stderr:+ pushd fsstress 2024-02-10T06:15:51.363 INFO:tasks.workunit.client.0.smithi089.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-10T06:15:51.363 INFO:tasks.workunit.client.0.smithi089.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-10T06:15:51.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:51 smithi089 ceph-mon[29217]: pgmap v219: 65 pgs: 65 active+clean; 8.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 683 KiB/s rd, 709 KiB/s wr, 4 op/s 2024-02-10T06:15:51.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:51 smithi173 ceph-mon[36251]: pgmap v219: 65 pgs: 65 active+clean; 8.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 683 KiB/s rd, 709 KiB/s wr, 4 op/s 2024-02-10T06:15:52.718 INFO:tasks.workunit.client.0.smithi089.stderr:+ tar xzf ltp-full.tgz 2024-02-10T06:15:53.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:53 smithi089 ceph-mon[29217]: pgmap v220: 65 pgs: 65 active+clean; 12 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 683 KiB/s rd, 1019 KiB/s wr, 59 op/s 2024-02-10T06:15:53.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:53 smithi173 ceph-mon[36251]: pgmap v220: 65 pgs: 65 active+clean; 12 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 683 KiB/s rd, 1019 KiB/s wr, 59 op/s 2024-02-10T06:15:55.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:55 smithi089 ceph-mon[29217]: pgmap v221: 65 pgs: 65 active+clean; 27 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-02-10T06:15:55.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:55 smithi173 ceph-mon[36251]: pgmap v221: 65 pgs: 65 active+clean; 27 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-02-10T06:15:57.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:57 smithi089 ceph-mon[29217]: pgmap v222: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 450 op/s 2024-02-10T06:15:57.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:57 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:57.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:57 smithi173 ceph-mon[36251]: pgmap v222: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 450 op/s 2024-02-10T06:15:57.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:57 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:59.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:15:59 smithi089 ceph-mon[29217]: pgmap v223: 65 pgs: 65 active+clean; 61 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 480 op/s 2024-02-10T06:15:59.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:15:59 smithi173 ceph-mon[36251]: pgmap v223: 65 pgs: 65 active+clean; 61 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 480 op/s 2024-02-10T06:16:01.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:01 smithi089 ceph-mon[29217]: pgmap v224: 65 pgs: 65 active+clean; 78 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 6.8 MiB/s wr, 558 op/s 2024-02-10T06:16:01.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:01 smithi173 ceph-mon[36251]: pgmap v224: 65 pgs: 65 active+clean; 78 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 6.8 MiB/s wr, 558 op/s 2024-02-10T06:16:03.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:03 smithi089 ceph-mon[29217]: pgmap v225: 65 pgs: 65 active+clean; 78 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 6.1 MiB/s wr, 556 op/s 2024-02-10T06:16:03.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:03 smithi173 ceph-mon[36251]: pgmap v225: 65 pgs: 65 active+clean; 78 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 6.1 MiB/s wr, 556 op/s 2024-02-10T06:16:05.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:05 smithi089 ceph-mon[29217]: pgmap v226: 65 pgs: 65 active+clean; 89 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 6.7 MiB/s wr, 507 op/s 2024-02-10T06:16:05.649 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:05 smithi173 ceph-mon[36251]: pgmap v226: 65 pgs: 65 active+clean; 89 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 6.7 MiB/s wr, 507 op/s 2024-02-10T06:16:07.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:07 smithi089 ceph-mon[29217]: pgmap v227: 65 pgs: 65 active+clean; 93 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.7 MiB/s wr, 339 op/s 2024-02-10T06:16:07.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:07 smithi173 ceph-mon[36251]: pgmap v227: 65 pgs: 65 active+clean; 93 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.7 MiB/s wr, 339 op/s 2024-02-10T06:16:09.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:09 smithi089 ceph-mon[29217]: pgmap v228: 65 pgs: 65 active+clean; 103 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 4.6 MiB/s wr, 303 op/s 2024-02-10T06:16:09.399 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:09 smithi173 ceph-mon[36251]: pgmap v228: 65 pgs: 65 active+clean; 103 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 4.6 MiB/s wr, 303 op/s 2024-02-10T06:16:11.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:11 smithi089 ceph-mon[29217]: pgmap v229: 65 pgs: 65 active+clean; 138 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.9 MiB/s wr, 799 op/s 2024-02-10T06:16:11.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:11 smithi173 ceph-mon[36251]: pgmap v229: 65 pgs: 65 active+clean; 138 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.9 MiB/s wr, 799 op/s 2024-02-10T06:16:12.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:12 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:12.650 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:12 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:13.312 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:13 smithi173 ceph-mon[36251]: pgmap v230: 65 pgs: 65 active+clean; 141 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 770 op/s 2024-02-10T06:16:13.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:13 smithi089 ceph-mon[29217]: pgmap v230: 65 pgs: 65 active+clean; 141 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 770 op/s 2024-02-10T06:16:15.418 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:15 smithi173 ceph-mon[36251]: Upgrade: Updating mgr.smithi173.swdirm 2024-02-10T06:16:15.418 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:15 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:16:15.418 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:15 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.swdirm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:16:15.419 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:15 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:16:15.419 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:15 smithi173 ceph-mon[36251]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:16:15.419 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:15 smithi173 ceph-mon[36251]: Deploying daemon mgr.smithi173.swdirm on smithi173 2024-02-10T06:16:15.419 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:15 smithi173 ceph-mon[36251]: pgmap v231: 65 pgs: 65 active+clean; 149 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.3 MiB/s wr, 774 op/s 2024-02-10T06:16:15.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:15 smithi089 ceph-mon[29217]: Upgrade: Updating mgr.smithi173.swdirm 2024-02-10T06:16:15.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:15 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' 2024-02-10T06:16:15.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:15 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.swdirm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T06:16:15.606 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:15 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T06:16:15.607 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:15 smithi089 ceph-mon[29217]: from='mgr.14210 172.21.15.89:0/4223798012' entity='mgr.smithi089.pwmrye' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:16:15.607 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:15 smithi089 ceph-mon[29217]: Deploying daemon mgr.smithi173.swdirm on smithi173 2024-02-10T06:16:15.607 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:15 smithi089 ceph-mon[29217]: pgmap v231: 65 pgs: 65 active+clean; 149 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.3 MiB/s wr, 774 op/s 2024-02-10T06:16:15.982 INFO:tasks.workunit.client.1.smithi173.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-10T06:16:15.994 INFO:tasks.workunit.client.1.smithi173.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-10T06:16:15.994 INFO:tasks.workunit.client.1.smithi173.stderr:+ make 2024-02-10T06:16:16.517 INFO:tasks.workunit.client.1.smithi173.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-10T06:16:17.400 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Feb 10 06:16:17 smithi173 ceph-mon[36251]: pgmap v232: 65 pgs: 65 active+clean; 162 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 6.5 MiB/s wr, 777 op/s 2024-02-10T06:16:17.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:16:17 smithi089 ceph-mon[29217]: pgmap v232: 65 pgs: 65 active+clean; 162 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 6.5 MiB/s wr, 777 op/s 2024-02-10T06:16:19.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.590+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/3521488855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b180fcef0 msgr2=0x7f0b180fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:19.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.590+0000 7f0b1d6f3700 1 --2- 172.21.15.89:0/3521488855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b180fcef0 0x7f0b180fd320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0b08004940 tx=0x7f0b08009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:19.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.590+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/3521488855 shutdown_connections 2024-02-10T06:16:19.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.590+0000 7f0b1d6f3700 1 --2- 172.21.15.89:0/3521488855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b180fe0e0 0x7f0b180fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:19.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.590+0000 7f0b1d6f3700 1 --2- 172.21.15.89:0/3521488855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b180fcef0 0x7f0b180fd320 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:19.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.590+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/3521488855 >> 172.21.15.89:0/3521488855 conn(0x7f0b180f8620 msgr2=0x7f0b180faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:19.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.591+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/3521488855 shutdown_connections 2024-02-10T06:16:19.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.591+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/3521488855 wait complete. 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.591+0000 7f0b1d6f3700 1 Processor -- start 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.591+0000 7f0b1d6f3700 1 -- start start 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.591+0000 7f0b1d6f3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b180fcef0 0x7f0b1810fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.591+0000 7f0b1d6f3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b180fe0e0 0x7f0b18110200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.591+0000 7f0b1d6f3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b18110f60 con 0x7f0b180fcef0 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.591+0000 7f0b1d6f3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b181110d0 con 0x7f0b180fe0e0 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.591+0000 7f0b16ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b180fcef0 0x7f0b1810fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.591+0000 7f0b16ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b180fcef0 0x7f0b1810fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38096/0 (socket says 172.21.15.89:38096) 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.591+0000 7f0b16ffd700 1 -- 172.21.15.89:0/1715428177 learned_addr learned my addr 172.21.15.89:0/1715428177 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.592+0000 7f0b16ffd700 1 -- 172.21.15.89:0/1715428177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b180fe0e0 msgr2=0x7f0b18110200 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.592+0000 7f0b16ffd700 1 --2- 172.21.15.89:0/1715428177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b180fe0e0 0x7f0b18110200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.592+0000 7f0b16ffd700 1 -- 172.21.15.89:0/1715428177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b08005040 con 0x7f0b180fcef0 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.592+0000 7f0b16ffd700 1 --2- 172.21.15.89:0/1715428177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b180fcef0 0x7f0b1810fcc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0b08000c00 tx=0x7f0b08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:19.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.592+0000 7f0b0f7fe700 1 -- 172.21.15.89:0/1715428177 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b0800e030 con 0x7f0b180fcef0 2024-02-10T06:16:19.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.592+0000 7f0b0f7fe700 1 -- 172.21.15.89:0/1715428177 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b08004290 con 0x7f0b180fcef0 2024-02-10T06:16:19.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.592+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/1715428177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b18111350 con 0x7f0b180fcef0 2024-02-10T06:16:19.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.592+0000 7f0b0f7fe700 1 -- 172.21.15.89:0/1715428177 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b0801c640 con 0x7f0b180fcef0 2024-02-10T06:16:19.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.592+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/1715428177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b18111890 con 0x7f0b180fcef0 2024-02-10T06:16:19.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.593+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/1715428177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b18102210 con 0x7f0b180fcef0 2024-02-10T06:16:19.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.594+0000 7f0b0f7fe700 1 -- 172.21.15.89:0/1715428177 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0b0800bbf0 con 0x7f0b180fcef0 2024-02-10T06:16:19.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.594+0000 7f0b0f7fe700 1 --2- 172.21.15.89:0/1715428177 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0b040692a0 0x7f0b0406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:19.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.594+0000 7f0b0f7fe700 1 -- 172.21.15.89:0/1715428177 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0b08088b90 con 0x7f0b180fcef0 2024-02-10T06:16:19.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.594+0000 7f0b0ffff700 1 --2- 172.21.15.89:0/1715428177 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0b040692a0 0x7f0b0406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:19.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.595+0000 7f0b0ffff700 1 --2- 172.21.15.89:0/1715428177 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0b040692a0 0x7f0b0406b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0b00009740 tx=0x7f0b00006c60 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:19.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.599+0000 7f0b0f7fe700 1 -- 172.21.15.89:0/1715428177 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0b080589a0 con 0x7f0b180fcef0 2024-02-10T06:16:19.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.796+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/1715428177 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b18111bc0 con 0x7f0b040692a0 2024-02-10T06:16:19.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.797+0000 7f0b0f7fe700 1 -- 172.21.15.89:0/1715428177 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7f0b18111bc0 con 0x7f0b040692a0 2024-02-10T06:16:19.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.799+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/1715428177 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0b040692a0 msgr2=0x7f0b0406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:19.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.799+0000 7f0b1d6f3700 1 --2- 172.21.15.89:0/1715428177 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0b040692a0 0x7f0b0406b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0b00009740 tx=0x7f0b00006c60 comp rx=0 tx=0).stop 2024-02-10T06:16:19.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.799+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/1715428177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b180fcef0 msgr2=0x7f0b1810fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:19.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.799+0000 7f0b1d6f3700 1 --2- 172.21.15.89:0/1715428177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b180fcef0 0x7f0b1810fcc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0b08000c00 tx=0x7f0b08009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:19.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.799+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/1715428177 shutdown_connections 2024-02-10T06:16:19.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.799+0000 7f0b1d6f3700 1 --2- 172.21.15.89:0/1715428177 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0b040692a0 0x7f0b0406b750 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:19.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.799+0000 7f0b1d6f3700 1 --2- 172.21.15.89:0/1715428177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b180fcef0 0x7f0b1810fcc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:19.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.799+0000 7f0b1d6f3700 1 --2- 172.21.15.89:0/1715428177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b180fe0e0 0x7f0b18110200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:19.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.799+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/1715428177 >> 172.21.15.89:0/1715428177 conn(0x7f0b180f8620 msgr2=0x7f0b180ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:19.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.799+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/1715428177 shutdown_connections 2024-02-10T06:16:19.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.799+0000 7f0b1d6f3700 1 -- 172.21.15.89:0/1715428177 wait complete. 2024-02-10T06:16:19.810 INFO:teuthology.orchestra.run.smithi089.stdout:true 2024-02-10T06:16:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.900+0000 7f57686c4700 1 -- 172.21.15.89:0/4229172264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57600fbe40 msgr2=0x7f57600fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.900+0000 7f57686c4700 1 --2- 172.21.15.89:0/4229172264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57600fbe40 0x7f57600fc250 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f575c004940 tx=0x7f575c009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.900+0000 7f57686c4700 1 -- 172.21.15.89:0/4229172264 shutdown_connections 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.900+0000 7f57686c4700 1 --2- 172.21.15.89:0/4229172264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57600fcfb0 0x7f57600fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.900+0000 7f57686c4700 1 --2- 172.21.15.89:0/4229172264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57600fbe40 0x7f57600fc250 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.900+0000 7f57686c4700 1 -- 172.21.15.89:0/4229172264 >> 172.21.15.89:0/4229172264 conn(0x7f57600f74e0 msgr2=0x7f57600f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.900+0000 7f57686c4700 1 -- 172.21.15.89:0/4229172264 shutdown_connections 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.900+0000 7f57686c4700 1 -- 172.21.15.89:0/4229172264 wait complete. 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f57686c4700 1 Processor -- start 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f57686c4700 1 -- start start 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f57686c4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57600fbe40 0x7f576010da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f57686c4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57600fcfb0 0x7f576010df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f57686c4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576010e5d0 con 0x7f57600fbe40 2024-02-10T06:16:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f57686c4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576010ee10 con 0x7f57600fcfb0 2024-02-10T06:16:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f5766460700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57600fbe40 0x7f576010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f5766460700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57600fbe40 0x7f576010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38104/0 (socket says 172.21.15.89:38104) 2024-02-10T06:16:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f5766460700 1 -- 172.21.15.89:0/4061325719 learned_addr learned my addr 172.21.15.89:0/4061325719 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:16:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f5766460700 1 -- 172.21.15.89:0/4061325719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57600fcfb0 msgr2=0x7f576010df40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-10T06:16:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f5766460700 1 --2- 172.21.15.89:0/4061325719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57600fcfb0 0x7f576010df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.901+0000 7f5766460700 1 -- 172.21.15.89:0/4061325719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c005040 con 0x7f57600fbe40 2024-02-10T06:16:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.902+0000 7f5766460700 1 --2- 172.21.15.89:0/4061325719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57600fbe40 0x7f576010da00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f575c008da0 tx=0x7f575c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.902+0000 7f5756ffd700 1 -- 172.21.15.89:0/4061325719 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f575c00e070 con 0x7f57600fbe40 2024-02-10T06:16:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.902+0000 7f5756ffd700 1 -- 172.21.15.89:0/4061325719 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f575c009430 con 0x7f57600fbe40 2024-02-10T06:16:19.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.902+0000 7f57686c4700 1 -- 172.21.15.89:0/4061325719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576010f090 con 0x7f57600fbe40 2024-02-10T06:16:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.902+0000 7f5756ffd700 1 -- 172.21.15.89:0/4061325719 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f575c01c400 con 0x7f57600fbe40 2024-02-10T06:16:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.902+0000 7f57686c4700 1 -- 172.21.15.89:0/4061325719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576010f570 con 0x7f57600fbe40 2024-02-10T06:16:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.903+0000 7f57686c4700 1 -- 172.21.15.89:0/4061325719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576004bed0 con 0x7f57600fbe40 2024-02-10T06:16:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.903+0000 7f5756ffd700 1 -- 172.21.15.89:0/4061325719 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f575c00bbf0 con 0x7f57600fbe40 2024-02-10T06:16:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.904+0000 7f5756ffd700 1 --2- 172.21.15.89:0/4061325719 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f574c0692b0 0x7f574c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.904+0000 7f5756ffd700 1 -- 172.21.15.89:0/4061325719 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f575c088a50 con 0x7f57600fbe40 2024-02-10T06:16:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.904+0000 7f5765c5f700 1 --2- 172.21.15.89:0/4061325719 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f574c0692b0 0x7f574c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.904+0000 7f5765c5f700 1 --2- 172.21.15.89:0/4061325719 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f574c0692b0 0x7f574c06b760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5750009740 tx=0x7f5750006c60 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:19.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:19.908+0000 7f5756ffd700 1 -- 172.21.15.89:0/4061325719 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f575c058740 con 0x7f57600fbe40 2024-02-10T06:16:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.114+0000 7f57686c4700 1 -- 172.21.15.89:0/4061325719 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f576010fc50 con 0x7f574c0692b0 2024-02-10T06:16:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.116+0000 7f5756ffd700 1 -- 172.21.15.89:0/4061325719 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7f576010fc50 con 0x7f574c0692b0 2024-02-10T06:16:20.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.118+0000 7f57686c4700 1 -- 172.21.15.89:0/4061325719 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f574c0692b0 msgr2=0x7f574c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:20.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.118+0000 7f57686c4700 1 --2- 172.21.15.89:0/4061325719 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f574c0692b0 0x7f574c06b760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5750009740 tx=0x7f5750006c60 comp rx=0 tx=0).stop 2024-02-10T06:16:20.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.118+0000 7f57686c4700 1 -- 172.21.15.89:0/4061325719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57600fbe40 msgr2=0x7f576010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:20.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.118+0000 7f57686c4700 1 --2- 172.21.15.89:0/4061325719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57600fbe40 0x7f576010da00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f575c008da0 tx=0x7f575c009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:20.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.118+0000 7f57686c4700 1 -- 172.21.15.89:0/4061325719 shutdown_connections 2024-02-10T06:16:20.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.118+0000 7f57686c4700 1 --2- 172.21.15.89:0/4061325719 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f574c0692b0 0x7f574c06b760 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.118+0000 7f57686c4700 1 --2- 172.21.15.89:0/4061325719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57600fbe40 0x7f576010da00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.118+0000 7f57686c4700 1 --2- 172.21.15.89:0/4061325719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57600fcfb0 0x7f576010df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.118+0000 7f57686c4700 1 -- 172.21.15.89:0/4061325719 >> 172.21.15.89:0/4061325719 conn(0x7f57600f74e0 msgr2=0x7f57600656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:20.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.118+0000 7f57686c4700 1 -- 172.21.15.89:0/4061325719 shutdown_connections 2024-02-10T06:16:20.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.118+0000 7f57686c4700 1 -- 172.21.15.89:0/4061325719 wait complete. 2024-02-10T06:16:20.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.217+0000 7f4420fe7700 1 -- 172.21.15.89:0/24107615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f441c0fe0e0 msgr2=0x7f441c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:20.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.217+0000 7f4420fe7700 1 --2- 172.21.15.89:0/24107615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f441c0fe0e0 0x7f441c0fe550 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4410004940 tx=0x7f4410009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:20.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 -- 172.21.15.89:0/24107615 shutdown_connections 2024-02-10T06:16:20.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 --2- 172.21.15.89:0/24107615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f441c0fe0e0 0x7f441c0fe550 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 --2- 172.21.15.89:0/24107615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f441c0fcef0 0x7f441c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 -- 172.21.15.89:0/24107615 >> 172.21.15.89:0/24107615 conn(0x7f441c0f8620 msgr2=0x7f441c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:20.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 -- 172.21.15.89:0/24107615 shutdown_connections 2024-02-10T06:16:20.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 -- 172.21.15.89:0/24107615 wait complete. 2024-02-10T06:16:20.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 Processor -- start 2024-02-10T06:16:20.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 -- start start 2024-02-10T06:16:20.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f441c0fcef0 0x7f441c10d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:20.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f441c0fe0e0 0x7f441c10df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:20.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c10e5c0 con 0x7f441c0fe0e0 2024-02-10T06:16:20.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4420fe7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c10ee00 con 0x7f441c0fcef0 2024-02-10T06:16:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4419d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f441c0fe0e0 0x7f441c10df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4419d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f441c0fe0e0 0x7f441c10df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38122/0 (socket says 172.21.15.89:38122) 2024-02-10T06:16:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.218+0000 7f4419d9b700 1 -- 172.21.15.89:0/1394234517 learned_addr learned my addr 172.21.15.89:0/1394234517 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:16:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.219+0000 7f4419d9b700 1 -- 172.21.15.89:0/1394234517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f441c0fcef0 msgr2=0x7f441c10d9f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-10T06:16:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.219+0000 7f4419d9b700 1 --2- 172.21.15.89:0/1394234517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f441c0fcef0 0x7f441c10d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.219+0000 7f4419d9b700 1 -- 172.21.15.89:0/1394234517 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4410005040 con 0x7f441c0fe0e0 2024-02-10T06:16:20.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.219+0000 7f4419d9b700 1 --2- 172.21.15.89:0/1394234517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f441c0fe0e0 0x7f441c10df30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4410000c00 tx=0x7f4410009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:20.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.219+0000 7f440affd700 1 -- 172.21.15.89:0/1394234517 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f441000e030 con 0x7f441c0fe0e0 2024-02-10T06:16:20.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.219+0000 7f440affd700 1 -- 172.21.15.89:0/1394234517 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4410004290 con 0x7f441c0fe0e0 2024-02-10T06:16:20.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.219+0000 7f4420fe7700 1 -- 172.21.15.89:0/1394234517 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c10f080 con 0x7f441c0fe0e0 2024-02-10T06:16:20.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.220+0000 7f440affd700 1 -- 172.21.15.89:0/1394234517 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f441001c640 con 0x7f441c0fe0e0 2024-02-10T06:16:20.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.220+0000 7f4420fe7700 1 -- 172.21.15.89:0/1394234517 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c10f5c0 con 0x7f441c0fe0e0 2024-02-10T06:16:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.221+0000 7f4420fe7700 1 -- 172.21.15.89:0/1394234517 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441c04bed0 con 0x7f441c0fe0e0 2024-02-10T06:16:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.221+0000 7f440affd700 1 -- 172.21.15.89:0/1394234517 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f441000bbf0 con 0x7f441c0fe0e0 2024-02-10T06:16:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.221+0000 7f440affd700 1 --2- 172.21.15.89:0/1394234517 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f44040692a0 0x7f440406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.221+0000 7f440affd700 1 -- 172.21.15.89:0/1394234517 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4410088960 con 0x7f441c0fe0e0 2024-02-10T06:16:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.221+0000 7f441a59c700 1 --2- 172.21.15.89:0/1394234517 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f44040692a0 0x7f440406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.222+0000 7f441a59c700 1 --2- 172.21.15.89:0/1394234517 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f44040692a0 0x7f440406b750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f441c0fcc30 tx=0x7f440c006c60 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:20.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.225+0000 7f440affd700 1 -- 172.21.15.89:0/1394234517 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4410058770 con 0x7f441c0fe0e0 2024-02-10T06:16:20.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.417+0000 7f4420fe7700 1 -- 172.21.15.89:0/1394234517 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f441c102ac0 con 0x7f44040692a0 2024-02-10T06:16:20.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.427+0000 7f440affd700 1 -- 172.21.15.89:0/1394234517 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f441c102ac0 con 0x7f44040692a0 2024-02-10T06:16:20.429 INFO:teuthology.orchestra.run.smithi089.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T06:16:20.429 INFO:teuthology.orchestra.run.smithi089.stdout:alertmanager.smithi089 smithi089 *:9093,9094 running (5m) 94s ago 7m 24.6M - 0.25.0 c8568f914cd2 667ad25cb367 2024-02-10T06:16:20.429 INFO:teuthology.orchestra.run.smithi089.stdout:crash.smithi089 smithi089 running (7m) 94s ago 7m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 e3aac8b18020 2024-02-10T06:16:20.429 INFO:teuthology.orchestra.run.smithi089.stdout:crash.smithi173 smithi173 running (6m) 98s ago 6m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 269616f4b479 2024-02-10T06:16:20.429 INFO:teuthology.orchestra.run.smithi089.stdout:grafana.smithi089 smithi089 *:3000 running (5m) 94s ago 6m 86.1M - 9.4.7 954c08fa6188 a6f72269ee53 2024-02-10T06:16:20.429 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi089.evydlq smithi089 running (111s) 94s ago 111s 14.2M - 17.2.7-505-g72ce7c1d a2a0bd126c96 9e2f6f4766e6 2024-02-10T06:16:20.429 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi089.mgkvmj smithi089 running (119s) 94s ago 118s 10.4M - 17.2.7-505-g72ce7c1d a2a0bd126c96 689a4ca9f11b 2024-02-10T06:16:20.429 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi173.cjkmdf smithi173 running (109s) 98s ago 108s 12.0M - 17.2.7-505-g72ce7c1d a2a0bd126c96 3d9eb72f6f12 2024-02-10T06:16:20.430 INFO:teuthology.orchestra.run.smithi089.stdout:mds.cephfs.smithi173.tumnkz smithi173 running (116s) 98s ago 115s 17.1M - 17.2.7-505-g72ce7c1d a2a0bd126c96 272e93861538 2024-02-10T06:16:20.430 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.smithi089.pwmrye smithi089 *:9283 running (8m) 94s ago 8m 481M - 17.2.7-505-g72ce7c1d a2a0bd126c96 1cd3521b2ebb 2024-02-10T06:16:20.430 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.smithi173.swdirm smithi173 *:8443,9283 running (6m) 98s ago 6m 433M - 17.2.7-505-g72ce7c1d a2a0bd126c96 6430108d8d2e 2024-02-10T06:16:20.430 INFO:teuthology.orchestra.run.smithi089.stdout:mon.smithi089 smithi089 running (8m) 94s ago 8m 46.7M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 0109d6119759 2024-02-10T06:16:20.430 INFO:teuthology.orchestra.run.smithi089.stdout:mon.smithi173 smithi173 running (6m) 98s ago 6m 35.8M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 08af8aefe37b 2024-02-10T06:16:20.430 INFO:teuthology.orchestra.run.smithi089.stdout:node-exporter.smithi089 smithi089 *:9100 running (7m) 94s ago 7m 14.7M - 1.5.0 0da6a335fe13 18b1f6a49022 2024-02-10T06:16:20.430 INFO:teuthology.orchestra.run.smithi089.stdout:node-exporter.smithi173 smithi173 *:9100 running (6m) 98s ago 6m 16.5M - 1.5.0 0da6a335fe13 f7d0b906813f 2024-02-10T06:16:20.430 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0 smithi089 running (5m) 94s ago 5m 47.5M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 e372a21351db 2024-02-10T06:16:20.430 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1 smithi089 running (5m) 94s ago 5m 46.3M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 42f862e2f6ca 2024-02-10T06:16:20.431 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2 smithi089 running (4m) 94s ago 4m 45.2M 4288M 17.2.7-505-g72ce7c1d a2a0bd126c96 69b16e6f89fb 2024-02-10T06:16:20.431 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3 smithi173 running (4m) 98s ago 4m 43.3M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 997c2081faec 2024-02-10T06:16:20.431 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4 smithi173 running (3m) 98s ago 3m 42.0M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 d3c6d0e77356 2024-02-10T06:16:20.431 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5 smithi173 running (3m) 98s ago 3m 41.2M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 bb845c14ca98 2024-02-10T06:16:20.431 INFO:teuthology.orchestra.run.smithi089.stdout:prometheus.smithi089 smithi089 *:9095 running (5m) 94s ago 6m 53.8M - 2.43.0 a07b618ecd1d 04539fe88109 2024-02-10T06:16:20.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.430+0000 7f4420fe7700 1 -- 172.21.15.89:0/1394234517 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f44040692a0 msgr2=0x7f440406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:20.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.430+0000 7f4420fe7700 1 --2- 172.21.15.89:0/1394234517 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f44040692a0 0x7f440406b750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f441c0fcc30 tx=0x7f440c006c60 comp rx=0 tx=0).stop 2024-02-10T06:16:20.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.430+0000 7f4420fe7700 1 -- 172.21.15.89:0/1394234517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f441c0fe0e0 msgr2=0x7f441c10df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:20.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.430+0000 7f4420fe7700 1 --2- 172.21.15.89:0/1394234517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f441c0fe0e0 0x7f441c10df30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4410000c00 tx=0x7f4410009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:20.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.430+0000 7f4420fe7700 1 -- 172.21.15.89:0/1394234517 shutdown_connections 2024-02-10T06:16:20.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.430+0000 7f4420fe7700 1 --2- 172.21.15.89:0/1394234517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f441c0fcef0 0x7f441c10d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.430+0000 7f4420fe7700 1 --2- 172.21.15.89:0/1394234517 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f44040692a0 0x7f440406b750 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.430+0000 7f4420fe7700 1 --2- 172.21.15.89:0/1394234517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f441c0fe0e0 0x7f441c10df30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.430+0000 7f4420fe7700 1 -- 172.21.15.89:0/1394234517 >> 172.21.15.89:0/1394234517 conn(0x7f441c0f8620 msgr2=0x7f441c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:20.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.430+0000 7f4420fe7700 1 -- 172.21.15.89:0/1394234517 shutdown_connections 2024-02-10T06:16:20.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.430+0000 7f4420fe7700 1 -- 172.21.15.89:0/1394234517 wait complete. 2024-02-10T06:16:20.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.529+0000 7fc0181a2700 1 -- 172.21.15.89:0/3021249080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100fbe40 msgr2=0x7fc0100fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:20.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.529+0000 7fc0181a2700 1 --2- 172.21.15.89:0/3021249080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100fbe40 0x7fc0100fc250 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc008004940 tx=0x7fc008009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:20.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.529+0000 7fc0181a2700 1 -- 172.21.15.89:0/3021249080 shutdown_connections 2024-02-10T06:16:20.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.529+0000 7fc0181a2700 1 --2- 172.21.15.89:0/3021249080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0100fcfb0 0x7fc0100fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.529+0000 7fc0181a2700 1 --2- 172.21.15.89:0/3021249080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100fbe40 0x7fc0100fc250 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.529+0000 7fc0181a2700 1 -- 172.21.15.89:0/3021249080 >> 172.21.15.89:0/3021249080 conn(0x7fc0100f74e0 msgr2=0x7fc0100f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:20.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.529+0000 7fc0181a2700 1 -- 172.21.15.89:0/3021249080 shutdown_connections 2024-02-10T06:16:20.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.529+0000 7fc0181a2700 1 -- 172.21.15.89:0/3021249080 wait complete. 2024-02-10T06:16:20.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.529+0000 7fc0181a2700 1 Processor -- start 2024-02-10T06:16:20.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.529+0000 7fc0181a2700 1 -- start start 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc0181a2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100fbe40 0x7fc01010da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc0181a2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0100fcfb0 0x7fc01010df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc0181a2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01010e5d0 con 0x7fc0100fbe40 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc0181a2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01010ee10 con 0x7fc0100fcfb0 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc015f3e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100fbe40 0x7fc01010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc015f3e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100fbe40 0x7fc01010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38142/0 (socket says 172.21.15.89:38142) 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc015f3e700 1 -- 172.21.15.89:0/29285394 learned_addr learned my addr 172.21.15.89:0/29285394 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc015f3e700 1 -- 172.21.15.89:0/29285394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0100fcfb0 msgr2=0x7fc01010df40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc015f3e700 1 --2- 172.21.15.89:0/29285394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0100fcfb0 0x7fc01010df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc015f3e700 1 -- 172.21.15.89:0/29285394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc008005040 con 0x7fc0100fbe40 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc015f3e700 1 --2- 172.21.15.89:0/29285394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100fbe40 0x7fc01010da00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc008004890 tx=0x7fc008009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc0067fc700 1 -- 172.21.15.89:0/29285394 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc00800e030 con 0x7fc0100fbe40 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc0067fc700 1 -- 172.21.15.89:0/29285394 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc008004290 con 0x7fc0100fbe40 2024-02-10T06:16:20.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.530+0000 7fc0181a2700 1 -- 172.21.15.89:0/29285394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01010f090 con 0x7fc0100fbe40 2024-02-10T06:16:20.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.531+0000 7fc0067fc700 1 -- 172.21.15.89:0/29285394 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc00801c660 con 0x7fc0100fbe40 2024-02-10T06:16:20.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.531+0000 7fc0181a2700 1 -- 172.21.15.89:0/29285394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01010f5d0 con 0x7fc0100fbe40 2024-02-10T06:16:20.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.531+0000 7fc0181a2700 1 -- 172.21.15.89:0/29285394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01004bed0 con 0x7fc0100fbe40 2024-02-10T06:16:20.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.532+0000 7fc0067fc700 1 -- 172.21.15.89:0/29285394 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc00800bbf0 con 0x7fc0100fbe40 2024-02-10T06:16:20.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.532+0000 7fc0067fc700 1 --2- 172.21.15.89:0/29285394 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbffc0692a0 0x7fbffc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:20.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.532+0000 7fc0067fc700 1 -- 172.21.15.89:0/29285394 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc008088c30 con 0x7fc0100fbe40 2024-02-10T06:16:20.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.532+0000 7fc01573d700 1 --2- 172.21.15.89:0/29285394 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbffc0692a0 0x7fbffc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:20.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.533+0000 7fc01573d700 1 --2- 172.21.15.89:0/29285394 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbffc0692a0 0x7fbffc06b750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc000009740 tx=0x7fc000006c60 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:20.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.537+0000 7fc0067fc700 1 -- 172.21.15.89:0/29285394 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc008058920 con 0x7fc0100fbe40 2024-02-10T06:16:20.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.794+0000 7fc0181a2700 1 -- 172.21.15.89:0/29285394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc01010fc40 con 0x7fc0100fbe40 2024-02-10T06:16:20.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.795+0000 7fc0067fc700 1 -- 172.21.15.89:0/29285394 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fc0080584b0 con 0x7fc0100fbe40 2024-02-10T06:16:20.796 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-10T06:16:20.796 INFO:teuthology.orchestra.run.smithi089.stdout: "mon": { 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": { 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": { 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 6 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": { 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 4 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-10T06:16:20.797 INFO:teuthology.orchestra.run.smithi089.stdout: "overall": { 2024-02-10T06:16:20.798 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 14 2024-02-10T06:16:20.798 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-10T06:16:20.798 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-10T06:16:20.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.797+0000 7fc0181a2700 1 -- 172.21.15.89:0/29285394 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbffc0692a0 msgr2=0x7fbffc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.797+0000 7fc0181a2700 1 --2- 172.21.15.89:0/29285394 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbffc0692a0 0x7fbffc06b750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc000009740 tx=0x7fc000006c60 comp rx=0 tx=0).stop 2024-02-10T06:16:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.797+0000 7fc0181a2700 1 -- 172.21.15.89:0/29285394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100fbe40 msgr2=0x7fc01010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.797+0000 7fc0181a2700 1 --2- 172.21.15.89:0/29285394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100fbe40 0x7fc01010da00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc008004890 tx=0x7fc008009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.797+0000 7fc0181a2700 1 -- 172.21.15.89:0/29285394 shutdown_connections 2024-02-10T06:16:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.797+0000 7fc0181a2700 1 --2- 172.21.15.89:0/29285394 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7fbffc0692a0 0x7fbffc06b750 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.797+0000 7fc0181a2700 1 --2- 172.21.15.89:0/29285394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0100fbe40 0x7fc01010da00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.797+0000 7fc0181a2700 1 --2- 172.21.15.89:0/29285394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0100fcfb0 0x7fc01010df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.797+0000 7fc0181a2700 1 -- 172.21.15.89:0/29285394 >> 172.21.15.89:0/29285394 conn(0x7fc0100f74e0 msgr2=0x7fc0100656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.797+0000 7fc0181a2700 1 -- 172.21.15.89:0/29285394 shutdown_connections 2024-02-10T06:16:20.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.797+0000 7fc0181a2700 1 -- 172.21.15.89:0/29285394 wait complete. 2024-02-10T06:16:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.896+0000 7f2abc930700 1 -- 172.21.15.89:0/837575906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab80fc7f0 msgr2=0x7f2ab80fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:20.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.896+0000 7f2abc930700 1 --2- 172.21.15.89:0/837575906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab80fc7f0 0x7f2ab80fcc40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8004940 tx=0x7f2aa8009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:20.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.896+0000 7f2abc930700 1 -- 172.21.15.89:0/837575906 shutdown_connections 2024-02-10T06:16:20.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.896+0000 7f2abc930700 1 --2- 172.21.15.89:0/837575906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab80fc7f0 0x7f2ab80fcc40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.896+0000 7f2abc930700 1 --2- 172.21.15.89:0/837575906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab80fb5e0 0x7f2ab80fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.896+0000 7f2abc930700 1 -- 172.21.15.89:0/837575906 >> 172.21.15.89:0/837575906 conn(0x7f2ab80f6ca0 msgr2=0x7f2ab80f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:20.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.896+0000 7f2abc930700 1 -- 172.21.15.89:0/837575906 shutdown_connections 2024-02-10T06:16:20.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.896+0000 7f2abc930700 1 -- 172.21.15.89:0/837575906 wait complete. 2024-02-10T06:16:20.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2abc930700 1 Processor -- start 2024-02-10T06:16:20.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2abc930700 1 -- start start 2024-02-10T06:16:20.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2abc930700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab80fb5e0 0x7f2ab8112bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:20.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2abc930700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab80fc7f0 0x7f2ab8115100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:20.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2abc930700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80fe6a0 con 0x7f2ab80fc7f0 2024-02-10T06:16:20.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2abc930700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80fe7e0 con 0x7f2ab80fb5e0 2024-02-10T06:16:20.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2ab5d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab80fc7f0 0x7f2ab8115100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:20.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2ab5d9b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab80fc7f0 0x7f2ab8115100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38174/0 (socket says 172.21.15.89:38174) 2024-02-10T06:16:20.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2ab5d9b700 1 -- 172.21.15.89:0/1728114935 learned_addr learned my addr 172.21.15.89:0/1728114935 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:16:20.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2ab5d9b700 1 -- 172.21.15.89:0/1728114935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab80fb5e0 msgr2=0x7f2ab8112bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-10T06:16:20.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2ab5d9b700 1 --2- 172.21.15.89:0/1728114935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab80fb5e0 0x7f2ab8112bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:20.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.897+0000 7f2ab5d9b700 1 -- 172.21.15.89:0/1728114935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8005040 con 0x7f2ab80fc7f0 2024-02-10T06:16:20.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.898+0000 7f2ab5d9b700 1 --2- 172.21.15.89:0/1728114935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab80fc7f0 0x7f2ab8115100 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800b3b0 tx=0x7f2aa8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:20.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.898+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1728114935 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa800e030 con 0x7f2ab80fc7f0 2024-02-10T06:16:20.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.898+0000 7f2abc930700 1 -- 172.21.15.89:0/1728114935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab8115640 con 0x7f2ab80fc7f0 2024-02-10T06:16:20.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.898+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1728114935 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2aa8013420 con 0x7f2ab80fc7f0 2024-02-10T06:16:20.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.898+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1728114935 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa800bee0 con 0x7f2ab80fc7f0 2024-02-10T06:16:20.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.898+0000 7f2abc930700 1 -- 172.21.15.89:0/1728114935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab8115aa0 con 0x7f2ab80fc7f0 2024-02-10T06:16:20.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.899+0000 7f2abc930700 1 -- 172.21.15.89:0/1728114935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab804bed0 con 0x7f2ab80fc7f0 2024-02-10T06:16:20.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.899+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1728114935 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2aa8013a70 con 0x7f2ab80fc7f0 2024-02-10T06:16:20.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.900+0000 7f2aa6ffd700 1 --2- 172.21.15.89:0/1728114935 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2aa00692a0 0x7f2aa006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:20.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.900+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1728114935 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2aa8088d10 con 0x7f2ab80fc7f0 2024-02-10T06:16:20.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.900+0000 7f2ab659c700 1 --2- 172.21.15.89:0/1728114935 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2aa00692a0 0x7f2aa006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:20.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.900+0000 7f2ab659c700 1 --2- 172.21.15.89:0/1728114935 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2aa00692a0 0x7f2aa006b750 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80fe430 tx=0x7f2aac006c60 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:20.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:20.904+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1728114935 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2aa8054f70 con 0x7f2ab80fc7f0 2024-02-10T06:16:21.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.129+0000 7f2abc930700 1 -- 172.21.15.89:0/1728114935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2ab8116000 con 0x7f2ab80fc7f0 2024-02-10T06:16:21.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.129+0000 7f2aa6ffd700 1 -- 172.21.15.89:0/1728114935 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1838 (secure 0 0 0) 0x7f2aa8058590 con 0x7f2ab80fc7f0 2024-02-10T06:16:21.130 INFO:teuthology.orchestra.run.smithi089.stdout:e15 2024-02-10T06:16:21.130 INFO:teuthology.orchestra.run.smithi089.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:legacy client fscid: 1 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:Filesystem 'cephfs' (1) 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:fs_name cephfs 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 15 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-02-10T06:14:18.236041+0000 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:modified 2024-02-10T06:14:36.018945+0000 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:tableserver 0 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:root 0 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:session_timeout 60 2024-02-10T06:16:21.131 INFO:teuthology.orchestra.run.smithi089.stdout:session_autoclose 300 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:max_file_size 1099511627776 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:required_client_features {} 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure 0 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure_osd_epoch 47 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:max_mds 1 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:in 0 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:up {0=24235} 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:failed 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:damaged 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:stopped 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:data_pools [3] 2024-02-10T06:16:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:metadata_pool 2 2024-02-10T06:16:21.133 INFO:teuthology.orchestra.run.smithi089.stdout:inline_data enabled 2024-02-10T06:16:21.133 INFO:teuthology.orchestra.run.smithi089.stdout:balancer 2024-02-10T06:16:21.133 INFO:teuthology.orchestra.run.smithi089.stdout:standby_count_wanted 1 2024-02-10T06:16:21.133 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi173.tumnkz{0:24235} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.173:6824/3924282139,v1:172.21.15.173:6825/3924282139] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:16:21.133 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi089.evydlq{0:14476} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.89:6828/3386828748,v1:172.21.15.89:6829/3386828748] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:16:21.133 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:16:21.133 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:16:21.133 INFO:teuthology.orchestra.run.smithi089.stdout:Standby daemons: 2024-02-10T06:16:21.133 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-10T06:16:21.133 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi173.cjkmdf{-1:24243} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.173:6826/2798078299,v1:172.21.15.173:6827/2798078299] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:16:21.133 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.cephfs.smithi089.mgkvmj{-1:24251} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.89:6826/3574254294,v1:172.21.15.89:6827/3574254294] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T06:16:21.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.131+0000 7f2abc930700 1 -- 172.21.15.89:0/1728114935 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2aa00692a0 msgr2=0x7f2aa006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:21.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.131+0000 7f2abc930700 1 --2- 172.21.15.89:0/1728114935 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2aa00692a0 0x7f2aa006b750 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80fe430 tx=0x7f2aac006c60 comp rx=0 tx=0).stop 2024-02-10T06:16:21.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.131+0000 7f2abc930700 1 -- 172.21.15.89:0/1728114935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab80fc7f0 msgr2=0x7f2ab8115100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:21.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.131+0000 7f2abc930700 1 --2- 172.21.15.89:0/1728114935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab80fc7f0 0x7f2ab8115100 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800b3b0 tx=0x7f2aa8009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:21.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.131+0000 7f2abc930700 1 -- 172.21.15.89:0/1728114935 shutdown_connections 2024-02-10T06:16:21.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.131+0000 7f2abc930700 1 --2- 172.21.15.89:0/1728114935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ab80fb5e0 0x7f2ab8112bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.131+0000 7f2abc930700 1 --2- 172.21.15.89:0/1728114935 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f2aa00692a0 0x7f2aa006b750 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.131+0000 7f2abc930700 1 --2- 172.21.15.89:0/1728114935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ab80fc7f0 0x7f2ab8115100 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.131+0000 7f2abc930700 1 -- 172.21.15.89:0/1728114935 >> 172.21.15.89:0/1728114935 conn(0x7f2ab80f6ca0 msgr2=0x7f2ab8100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:21.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.132+0000 7f2abc930700 1 -- 172.21.15.89:0/1728114935 shutdown_connections 2024-02-10T06:16:21.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.132+0000 7f2abc930700 1 -- 172.21.15.89:0/1728114935 wait complete. 2024-02-10T06:16:21.135 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 15 2024-02-10T06:16:21.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.232+0000 7ff4ada1f700 1 -- 172.21.15.89:0/3716543610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4a80fbe40 msgr2=0x7ff4a80fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:21.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.232+0000 7ff4ada1f700 1 --2- 172.21.15.89:0/3716543610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4a80fbe40 0x7ff4a80fc250 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009ce0 tx=0x7ff4a0004d10 comp rx=0 tx=0).stop 2024-02-10T06:16:21.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.232+0000 7ff4ada1f700 1 -- 172.21.15.89:0/3716543610 shutdown_connections 2024-02-10T06:16:21.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.232+0000 7ff4ada1f700 1 --2- 172.21.15.89:0/3716543610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4a80fcfb0 0x7ff4a80fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.232+0000 7ff4ada1f700 1 --2- 172.21.15.89:0/3716543610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4a80fbe40 0x7ff4a80fc250 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.232+0000 7ff4ada1f700 1 -- 172.21.15.89:0/3716543610 >> 172.21.15.89:0/3716543610 conn(0x7ff4a80f74e0 msgr2=0x7ff4a80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:21.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.232+0000 7ff4ada1f700 1 -- 172.21.15.89:0/3716543610 shutdown_connections 2024-02-10T06:16:21.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.232+0000 7ff4ada1f700 1 -- 172.21.15.89:0/3716543610 wait complete. 2024-02-10T06:16:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.232+0000 7ff4ada1f700 1 Processor -- start 2024-02-10T06:16:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.232+0000 7ff4ada1f700 1 -- start start 2024-02-10T06:16:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4ada1f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4a80fbe40 0x7ff4a810c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4ada1f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4a80fcfb0 0x7ff4a810ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4ada1f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a810d4e0 con 0x7ff4a80fcfb0 2024-02-10T06:16:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4ada1f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a810dd20 con 0x7ff4a80fbe40 2024-02-10T06:16:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4a67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4a80fcfb0 0x7ff4a810ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4a67fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4a80fcfb0 0x7ff4a810ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38202/0 (socket says 172.21.15.89:38202) 2024-02-10T06:16:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4a67fc700 1 -- 172.21.15.89:0/1806247082 learned_addr learned my addr 172.21.15.89:0/1806247082 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:16:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4a67fc700 1 -- 172.21.15.89:0/1806247082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4a80fbe40 msgr2=0x7ff4a810c910 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-10T06:16:21.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4a67fc700 1 --2- 172.21.15.89:0/1806247082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4a80fbe40 0x7ff4a810c910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4a67fc700 1 -- 172.21.15.89:0/1806247082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a0009940 con 0x7ff4a80fcfb0 2024-02-10T06:16:21.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4a67fc700 1 --2- 172.21.15.89:0/1806247082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4a80fcfb0 0x7ff4a810ce50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff49000b580 tx=0x7ff4900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:21.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff49f7fe700 1 -- 172.21.15.89:0/1806247082 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff49000f6f0 con 0x7ff4a80fcfb0 2024-02-10T06:16:21.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff49f7fe700 1 -- 172.21.15.89:0/1806247082 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff49000fd20 con 0x7ff4a80fcfb0 2024-02-10T06:16:21.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4ada1f700 1 -- 172.21.15.89:0/1806247082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a810e000 con 0x7ff4a80fcfb0 2024-02-10T06:16:21.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff49f7fe700 1 -- 172.21.15.89:0/1806247082 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff490011ea0 con 0x7ff4a80fcfb0 2024-02-10T06:16:21.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.233+0000 7ff4ada1f700 1 -- 172.21.15.89:0/1806247082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a810e540 con 0x7ff4a80fcfb0 2024-02-10T06:16:21.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.234+0000 7ff4ada1f700 1 -- 172.21.15.89:0/1806247082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a804bed0 con 0x7ff4a80fcfb0 2024-02-10T06:16:21.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.235+0000 7ff49f7fe700 1 -- 172.21.15.89:0/1806247082 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff49000f850 con 0x7ff4a80fcfb0 2024-02-10T06:16:21.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.235+0000 7ff49f7fe700 1 --2- 172.21.15.89:0/1806247082 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4940692a0 0x7ff49406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:21.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.236+0000 7ff49f7fe700 1 -- 172.21.15.89:0/1806247082 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff490086eb0 con 0x7ff4a80fcfb0 2024-02-10T06:16:21.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.236+0000 7ff4a6ffd700 1 --2- 172.21.15.89:0/1806247082 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4940692a0 0x7ff49406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:21.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.236+0000 7ff4a6ffd700 1 --2- 172.21.15.89:0/1806247082 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4940692a0 0x7ff49406b750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0000f80 tx=0x7ff4a0004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:21.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.240+0000 7ff49f7fe700 1 -- 172.21.15.89:0/1806247082 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff490056c20 con 0x7ff4a80fcfb0 2024-02-10T06:16:21.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.438+0000 7ff4ada1f700 1 -- 172.21.15.89:0/1806247082 --> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4a810eab0 con 0x7ff4940692a0 2024-02-10T06:16:21.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.439+0000 7ff49f7fe700 1 -- 172.21.15.89:0/1806247082 <== mgr.14210 v2:172.21.15.89:6800/223399833 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7ff4a810eab0 con 0x7ff4940692a0 2024-02-10T06:16:21.440 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-10T06:16:21.440 INFO:teuthology.orchestra.run.smithi089.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T06:16:21.440 INFO:teuthology.orchestra.run.smithi089.stdout: "in_progress": true, 2024-02-10T06:16:21.440 INFO:teuthology.orchestra.run.smithi089.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T06:16:21.440 INFO:teuthology.orchestra.run.smithi089.stdout: "services_complete": [], 2024-02-10T06:16:21.440 INFO:teuthology.orchestra.run.smithi089.stdout: "progress": "0/21 daemons upgraded", 2024-02-10T06:16:21.440 INFO:teuthology.orchestra.run.smithi089.stdout: "message": "Currently upgrading mgr daemons", 2024-02-10T06:16:21.440 INFO:teuthology.orchestra.run.smithi089.stdout: "is_paused": false 2024-02-10T06:16:21.440 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-10T06:16:21.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.441+0000 7ff4ada1f700 1 -- 172.21.15.89:0/1806247082 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4940692a0 msgr2=0x7ff49406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:21.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.441+0000 7ff4ada1f700 1 --2- 172.21.15.89:0/1806247082 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4940692a0 0x7ff49406b750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0000f80 tx=0x7ff4a0004d10 comp rx=0 tx=0).stop 2024-02-10T06:16:21.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.441+0000 7ff4ada1f700 1 -- 172.21.15.89:0/1806247082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4a80fcfb0 msgr2=0x7ff4a810ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:21.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.441+0000 7ff4ada1f700 1 --2- 172.21.15.89:0/1806247082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4a80fcfb0 0x7ff4a810ce50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff49000b580 tx=0x7ff4900091a0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.441+0000 7ff4ada1f700 1 -- 172.21.15.89:0/1806247082 shutdown_connections 2024-02-10T06:16:21.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.441+0000 7ff4ada1f700 1 --2- 172.21.15.89:0/1806247082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4a80fbe40 0x7ff4a810c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.441+0000 7ff4ada1f700 1 --2- 172.21.15.89:0/1806247082 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7ff4940692a0 0x7ff49406b750 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.441+0000 7ff4ada1f700 1 --2- 172.21.15.89:0/1806247082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4a80fcfb0 0x7ff4a810ce50 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.441+0000 7ff4ada1f700 1 -- 172.21.15.89:0/1806247082 >> 172.21.15.89:0/1806247082 conn(0x7ff4a80f74e0 msgr2=0x7ff4a8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:21.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.441+0000 7ff4ada1f700 1 -- 172.21.15.89:0/1806247082 shutdown_connections 2024-02-10T06:16:21.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.441+0000 7ff4ada1f700 1 -- 172.21.15.89:0/1806247082 wait complete. 2024-02-10T06:16:21.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.541+0000 7f0e824f4700 1 -- 172.21.15.89:0/2667812085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e7c0fe2b0 msgr2=0x7f0e7c0fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:21.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.541+0000 7f0e824f4700 1 --2- 172.21.15.89:0/2667812085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e7c0fe2b0 0x7f0e7c0fe700 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0e70004940 tx=0x7f0e70009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:21.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.541+0000 7f0e824f4700 1 -- 172.21.15.89:0/2667812085 shutdown_connections 2024-02-10T06:16:21.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.541+0000 7f0e824f4700 1 --2- 172.21.15.89:0/2667812085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e7c0fe2b0 0x7f0e7c0fe700 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.541+0000 7f0e824f4700 1 --2- 172.21.15.89:0/2667812085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e7c0fb220 0x7f0e7c0fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.541+0000 7f0e824f4700 1 -- 172.21.15.89:0/2667812085 >> 172.21.15.89:0/2667812085 conn(0x7f0e7c0f6ca0 msgr2=0x7f0e7c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:21.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.541+0000 7f0e824f4700 1 -- 172.21.15.89:0/2667812085 shutdown_connections 2024-02-10T06:16:21.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.541+0000 7f0e824f4700 1 -- 172.21.15.89:0/2667812085 wait complete. 2024-02-10T06:16:21.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.542+0000 7f0e824f4700 1 Processor -- start 2024-02-10T06:16:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.542+0000 7f0e824f4700 1 -- start start 2024-02-10T06:16:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.542+0000 7f0e824f4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e7c0fb220 0x7f0e7c0fdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.542+0000 7f0e824f4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e7c0fe2b0 0x7f0e7c0fc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.542+0000 7f0e824f4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c0fc850 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.542+0000 7f0e824f4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c0fc9c0 con 0x7f0e7c0fb220 2024-02-10T06:16:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.542+0000 7f0e7b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e7c0fe2b0 0x7f0e7c0fc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.542+0000 7f0e7b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e7c0fe2b0 0x7f0e7c0fc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38216/0 (socket says 172.21.15.89:38216) 2024-02-10T06:16:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.542+0000 7f0e7b7fe700 1 -- 172.21.15.89:0/1140351889 learned_addr learned my addr 172.21.15.89:0/1140351889 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-10T06:16:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.542+0000 7f0e7b7fe700 1 -- 172.21.15.89:0/1140351889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e7c0fb220 msgr2=0x7f0e7c0fdd40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-10T06:16:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.542+0000 7f0e7b7fe700 1 --2- 172.21.15.89:0/1140351889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e7c0fb220 0x7f0e7c0fdd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.543+0000 7f0e7b7fe700 1 -- 172.21.15.89:0/1140351889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e70005040 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.543+0000 7f0e7b7fe700 1 --2- 172.21.15.89:0/1140351889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e7c0fe2b0 0x7f0e7c0fc1c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000b3b0 tx=0x7f0e70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.543+0000 7f0e78ff9700 1 -- 172.21.15.89:0/1140351889 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e7000e030 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.543+0000 7f0e78ff9700 1 -- 172.21.15.89:0/1140351889 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e70013420 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.543+0000 7f0e78ff9700 1 -- 172.21.15.89:0/1140351889 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e7000bee0 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.543+0000 7f0e824f4700 1 -- 172.21.15.89:0/1140351889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7c0fcc40 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.543+0000 7f0e824f4700 1 -- 172.21.15.89:0/1140351889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7c115af0 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.544+0000 7f0e824f4700 1 -- 172.21.15.89:0/1140351889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7c04bed0 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.544+0000 7f0e78ff9700 1 -- 172.21.15.89:0/1140351889 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0e70013a70 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.545+0000 7f0e78ff9700 1 --2- 172.21.15.89:0/1140351889 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e640692a0 0x7f0e6406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.545+0000 7f0e78ff9700 1 -- 172.21.15.89:0/1140351889 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0e70088d00 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.545+0000 7f0e7bfff700 1 --2- 172.21.15.89:0/1140351889 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e640692a0 0x7f0e6406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:16:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.545+0000 7f0e7bfff700 1 --2- 172.21.15.89:0/1140351889 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e640692a0 0x7f0e6406b750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0fdad0 tx=0x7f0e6c006c60 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:16:21.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.549+0000 7f0e78ff9700 1 -- 172.21.15.89:0/1140351889 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0e7008c030 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.801+0000 7f0e824f4700 1 -- 172.21.15.89:0/1140351889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0e7c115ee0 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.801+0000 7f0e78ff9700 1 -- 172.21.15.89:0/1140351889 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f0e700589f0 con 0x7f0e7c0fe2b0 2024-02-10T06:16:21.802 INFO:teuthology.orchestra.run.smithi089.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-10T06:16:21.802 INFO:teuthology.orchestra.run.smithi089.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-10T06:16:21.802 INFO:teuthology.orchestra.run.smithi089.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-10T06:16:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.803+0000 7f0e824f4700 1 -- 172.21.15.89:0/1140351889 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e640692a0 msgr2=0x7f0e6406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.803+0000 7f0e824f4700 1 --2- 172.21.15.89:0/1140351889 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e640692a0 0x7f0e6406b750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0fdad0 tx=0x7f0e6c006c60 comp rx=0 tx=0).stop 2024-02-10T06:16:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.803+0000 7f0e824f4700 1 -- 172.21.15.89:0/1140351889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e7c0fe2b0 msgr2=0x7f0e7c0fc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:16:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.803+0000 7f0e824f4700 1 --2- 172.21.15.89:0/1140351889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e7c0fe2b0 0x7f0e7c0fc1c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000b3b0 tx=0x7f0e70009e30 comp rx=0 tx=0).stop 2024-02-10T06:16:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.803+0000 7f0e824f4700 1 -- 172.21.15.89:0/1140351889 shutdown_connections 2024-02-10T06:16:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.803+0000 7f0e824f4700 1 --2- 172.21.15.89:0/1140351889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e7c0fb220 0x7f0e7c0fdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.803+0000 7f0e824f4700 1 --2- 172.21.15.89:0/1140351889 >> [v2:172.21.15.89:6800/223399833,v1:172.21.15.89:6801/223399833] conn(0x7f0e640692a0 0x7f0e6406b750 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.803+0000 7f0e824f4700 1 --2- 172.21.15.89:0/1140351889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e7c0fe2b0 0x7f0e7c0fc1c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:16:21.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.803+0000 7f0e824f4700 1 -- 172.21.15.89:0/1140351889 >> 172.21.15.89:0/1140351889 conn(0x7f0e7c0f6ca0 msgr2=0x7f0e7c1014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:16:21.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.804+0000 7f0e824f4700 1 -- 172.21.15.89:0/1140351889 shutdown_connections 2024-02-10T06:16:21.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:16:21.804+0000 7f0e824f4700 1 -- 172.21.15.89:0/1140351889 wait complete. 2024-02-10T06:16:39.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:39.266+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:39.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:39.266+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:39.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:39.266+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:40.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:40.282+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:40.282+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:40.282+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:40.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:40.292+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:40.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:40.292+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:40.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:40.292+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:41.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:41.273+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:41.273+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:41.273+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:41.320+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:41.320+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:41.321+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:42.239+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:42.239+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:42.239+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:42.319+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:42.319+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:42.319+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:43.042+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:43.042+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:43.042+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:43.368 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:43.267+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:43.369 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:43.267+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:43.369 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:43.267+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:43.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:43.368+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:43.368+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:43.368+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:44.285 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:44.032+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:44.286 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:44.032+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:44.286 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:44.032+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:44.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:44.285+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:44.285+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:44.285+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:44.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:44.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:44.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:44.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:44.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:44.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:45.326 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:45.053+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:45.326 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:45.053+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:45.327 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:45.053+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:45.325+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:45.325+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:45.325+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:45.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:45.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:45.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:45.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:45.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:45.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:46.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:46.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:46.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:46.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:46.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:46.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:46.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:46.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:46.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:46.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:46.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:46.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:46.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:46.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:46.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:46.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:47.389 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:47.328+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:47.390 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:47.328+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:47.390 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:47.328+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:47.390 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:47.118+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:47.390 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:47.118+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:47.390 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:47.118+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:47.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:47.388+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:47.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:47.388+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:47.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:47.388+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:48.326 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:48.086+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:48.327 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:48.086+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:48.327 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:48.086+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:48.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:48.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:48.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:48.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:48.362+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:48.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:48.362+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:48.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:48.362+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:49.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:49.121+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:49.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:49.122+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:49.122+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:49.352+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:49.352+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:49.352+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:49.341+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:49.341+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:49.341+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:50.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:50.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:50.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:50.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:50.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:50.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:50.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:50.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:50.321+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:50.321+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:50.321+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:51.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:51.157+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:51.157+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:51.157+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:51.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:51.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:51.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:51.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:51.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:51.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:51.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:51.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:51.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:52.407 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:52.133+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:52.407 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:52.133+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:52.407 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:52.133+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:52.408 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:52.369+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:52.408 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:52.369+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:52.408 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:52.369+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:52.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:52.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:52.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:53.406 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:53.147+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:53.406 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:53.147+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:53.407 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:53.147+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:53.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:53.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:53.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:53.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:53.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:53.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:54.390 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:54.120+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:54.390 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:54.120+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:54.390 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:54.120+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:54.394+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:54.394+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:54.394+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:54.388+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:54.388+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:54.388+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:55.399 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:55.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:55.399 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:55.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:55.399 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:55.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:55.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:55.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:55.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:55.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:55.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:55.438+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:55.438+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:55.438+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:56.417+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:56.417+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:56.417+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:56.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:56.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:56.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:56.178+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:56.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:56.178+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:56.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:56.178+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:57.461 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:57.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:57.461 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:57.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:57.461 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:57.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:57.461 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:57.202+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:57.462 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:57.202+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:57.462 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:57.202+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:57.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:57.460+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:57.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:57.460+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:57.460+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:58.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:58.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:58.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:58.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:58.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:58.498+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:58.498+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:58.498+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:58.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:58.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:58.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:59.471 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:59.215+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:59.471 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:59.215+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:59.471 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:16:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:16:59.215+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:16:59.472 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:59.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:59.472 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:59.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:59.472 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:16:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:16:59.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:16:59.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:59.470+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:59.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:59.470+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:16:59.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:16:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:16:59.470+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:00.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:00.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:00.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:00.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:00.371+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:00.371+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:00.371+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:00.465+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:00.465+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:00.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:00.465+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:01.508 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:01.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:01.508 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:01.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:01.509 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:01.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:01.509 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:01.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:01.509 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:01.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:01.509 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:01.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:01.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:01.507+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:01.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:01.508+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:01.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:01.508+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:02.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:02.267+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:02.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:02.267+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:02.267+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:02.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:02.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:02.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:02.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:02.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:02.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:02.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:02.413+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:02.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:02.413+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:02.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:02.413+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:03.447 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:03.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:03.448 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:03.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:03.448 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:03.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:03.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:03.513+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:03.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:03.513+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:03.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:03.513+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:03.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:03.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:03.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:04.444+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:04.444+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:04.444+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:04.307+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:04.307+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:04.307+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:04.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:04.551+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:04.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:04.551+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:04.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:04.551+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:05.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:05.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:05.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:05.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:05.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:05.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:05.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:05.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:05.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:05.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:05.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:05.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:05.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:06.572 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:06.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:06.572 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:06.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:06.573 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:06.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:06.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:06.297+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:06.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:06.297+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:06.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:06.297+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:06.571+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:06.571+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:06.571+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:07.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:07.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:07.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:07.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:07.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:07.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:07.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:07.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:07.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:07.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:07.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:07.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:07.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:07.580+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:07.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:07.580+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:07.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:07.580+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:08.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:08.466+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:08.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:08.466+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:08.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:08.466+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:08.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:08.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:08.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:08.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:08.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:08.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:09.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:08.628+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:09.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:08.628+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:09.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:08.628+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:09.583 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:09.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:09.583 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:09.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:09.583 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:09.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:09.583 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:09.289+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:09.584 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:09.289+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:09.584 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:09.289+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:09.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:09.582+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:09.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:09.582+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:09.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:09.582+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:10.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:10.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:10.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:10.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:10.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:10.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:10.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:10.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:10.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:11.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:10.626+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:11.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:10.626+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:11.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:10.626+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:11.477 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:11.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:11.477 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:11.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:11.478 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:11.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:11.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:11.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:11.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:11.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:11.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:11.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:11.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:11.657+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:11.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:11.657+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:11.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:11.657+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:12.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:12.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:12.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:12.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:12.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:12.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:12.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:12.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:12.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:13.039 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:12.699+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:13.039 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:12.699+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:13.040 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:12.699+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:13.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:13.237+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:13.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:13.237+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:13.237+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:13.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:13.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:13.459+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:13.459+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:13.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:13.697+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:13.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:13.697+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:13.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:13.697+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:14.481 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:14.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:14.481 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:14.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:14.481 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:14.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:14.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:14.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:14.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:14.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:14.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:14.672+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:14.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:14.672+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:14.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:14.672+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:15.473 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:15.167+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:15.473 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:15.167+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:15.473 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:15.167+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:15.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:15.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:15.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:15.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:15.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:15.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:15.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:15.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:15.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:15.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:15.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:16.517 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:16.173+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:16.518 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:16.173+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:16.518 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:16.173+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:16.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:16.894 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:16.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:16.894 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:16.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:16.894 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:16.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:16.895 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:16.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:16.895 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:16.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:17.509 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:17.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:17.510 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:17.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:17.510 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:17.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:17.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:17.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:17.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:17.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:17.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:17.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:17.642+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:17.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:17.642+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:17.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:17.642+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:18.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:18.170+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:18.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:18.170+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:18.170+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:18.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:18.655+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:18.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:18.655+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:18.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:18.655+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:18.538+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:18.538+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:18.538+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:19.561 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:19.178+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:19.561 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:19.178+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:19.561 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:19.178+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:19.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:19.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:19.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:19.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:19.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:19.636+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:19.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:19.636+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:19.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:19.636+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:20.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:20.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:20.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:20.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:20.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:20.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:20.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:20.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:20.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:20.678+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:20.678+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:20.678+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:21.549 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:21.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:21.549 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:21.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:21.549 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:21.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:21.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:21.548+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:21.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:21.548+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:21.548+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:21.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:21.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:21.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:21.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:21.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:21.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:22.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:22.222+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:22.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:22.222+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:22.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:22.222+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:22.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:22.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:22.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:22.630+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:22.630+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:22.630+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:23.512 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:23.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:23.512 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:23.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:23.512 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:23.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:23.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:23.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:23.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:23.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:23.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:23.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:23.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:23.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:23.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:23.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:24.497 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:24.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:24.497 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:24.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:24.497 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:24.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:24.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:24.496+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:24.496+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:24.496+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:24.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:24.657+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:24.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:24.657+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:24.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:24.657+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:25.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:25.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:25.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:25.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:25.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:25.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:25.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:25.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:25.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:25.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:25.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:25.622+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:25.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:25.622+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:25.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:25.622+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:26.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:26.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:26.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:26.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:26.158+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:26.158+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:26.158+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:27.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:26.638+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:27.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:26.638+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:26.638+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:27.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:27.426+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:27.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:27.426+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:27.426+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:27.199+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:27.199+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:27.199+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:28.035 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:27.678+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:28.035 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:27.678+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:28.036 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:27.678+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:28.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:28.153+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:28.153+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:28.153+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:28.701 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:28.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:28.701 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:28.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:28.701 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:28.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:29.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:28.700+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:28.700+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:28.700+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:29.483 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:29.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:29.483 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:29.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:29.483 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:29.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:29.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:29.671+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:29.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:29.671+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:29.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:29.671+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:29.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:29.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:29.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:30.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:30.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:30.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:30.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:30.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:30.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:30.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:30.639+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:30.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:30.639+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:30.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:30.639+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:30.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:30.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:30.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:31.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:31.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:31.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:31.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:31.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:31.471+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:31.471+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:31.471+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:31.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:31.616+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:31.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:31.616+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:31.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:31.616+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:32.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:32.098+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:32.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:32.099+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:32.099+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:32.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:32.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:32.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:32.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:32.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:32.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:32.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:32.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:32.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:32.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:32.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:32.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:33.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:33.107+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:33.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:33.107+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:33.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:33.107+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:33.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:33.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:33.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:33.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:33.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:33.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:33.625+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:33.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:33.625+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:33.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:33.625+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:34.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:34.061+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:34.061+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:34.061+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:34.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:34.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:34.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:34.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:34.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:34.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:34.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:34.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:34.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:34.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:35.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:35.029+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:35.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:35.029+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:35.029+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:35.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:35.475+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:35.475+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:35.475+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:35.710+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:35.710+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:35.710+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:36.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:36.000+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:36.000+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:36.000+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:36.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:36.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:36.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:36.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:36.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:36.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:36.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:36.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:37.033+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:37.033+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:37.033+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:37.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:37.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:37.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:37.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:37.732+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:37.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:37.732+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:37.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:37.732+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:38.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:37.987+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:38.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:37.987+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:38.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:37.987+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:38.577+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:38.577+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:38.577+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:38.688+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:38.688+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:38.688+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:39.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:39.010+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:39.010+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:39.010+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:40.032 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:39.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:40.033 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:39.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:40.033 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:39.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:40.033 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:39.723+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:40.033 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:39.723+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:40.033 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:39.723+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:40.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:40.030+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:40.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:40.030+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:40.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:40.030+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:41.019 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:40.657+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:41.019 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:40.657+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:41.019 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:40.657+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:41.019 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:40.731+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:41.019 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:40.731+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:41.019 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:40.731+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:41.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:41.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:41.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:41.995 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:41.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:41.995 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:41.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:41.995 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:41.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:41.995 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:41.745+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:41.995 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:41.745+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:41.995 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:41.745+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:42.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:41.994+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:41.994+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:41.994+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:43.040 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:42.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:43.040 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:42.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:43.040 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:42.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:43.041 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:42.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:43.041 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:42.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:43.041 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:42.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:43.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:43.039+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:43.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:43.039+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:43.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:43.039+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:44.059 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:43.736+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:44.059 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:43.736+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:44.059 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:43.736+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:44.060 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:43.765+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:44.060 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:43.765+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:44.060 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:43.765+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:44.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:44.058+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:44.058+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:44.058+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:45.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:44.740+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:45.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:44.740+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:45.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:44.740+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:45.098 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:44.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:45.098 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:44.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:45.098 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:44.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:45.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:45.097+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:45.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:45.097+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:45.097+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:46.080 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:45.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:46.080 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:45.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:46.080 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:45.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:46.080 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:45.768+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:46.080 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:45.768+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:46.080 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:45.768+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:46.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:46.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:46.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:46.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:46.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:46.079+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:46.807 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:46.701+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:46.807 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:46.701+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:46.807 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:46.701+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:47.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:47.054+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:47.054+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:47.054+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:47.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:46.806+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:47.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:46.806+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:47.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:46.806+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:48.039 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:47.831+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:48.039 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:47.831+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:48.039 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:47.831+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:48.039 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:47.670+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:48.039 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:47.670+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:48.039 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:47.670+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:48.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:48.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:48.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:48.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:48.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:48.038+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:49.065 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:48.855+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:49.065 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:48.855+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:49.065 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:48.855+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:49.065 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:48.676+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:49.065 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:48.676+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:49.065 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:48.676+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:49.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:49.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:49.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:49.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:49.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:49.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:50.086 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:49.834+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:50.086 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:49.834+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:50.086 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:49.834+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:50.086 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:49.639+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:50.086 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:49.639+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:50.086 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:49.639+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:50.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:50.085+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:50.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:50.085+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:50.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:50.085+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:51.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:50.835+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:51.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:50.835+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:50.835+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:50.642+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:50.642+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:50.642+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:51.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:51.125+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:51.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:51.125+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:51.125+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:52.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:51.626+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:52.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:51.626+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:52.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:51.626+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:51.812+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:51.812+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:51.812+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:52.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:52.121+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:52.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:52.121+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:52.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:52.121+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:52.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:52.851+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:52.851+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:52.851+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:52.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:52.603+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:52.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:52.603+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:52.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:52.603+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:53.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:53.131+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:53.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:53.131+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:53.131+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:53.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:53.563+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:53.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:53.563+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:53.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:53.563+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:54.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:53.802+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:54.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:53.802+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:54.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:53.803+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:54.545 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:54.152+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:54.546 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:54.152+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:54.546 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:54.152+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:54.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:54.765+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:54.765+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:54.765+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:54.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:54.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:54.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:55.562 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:55.157+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:55.562 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:55.157+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:55.563 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:55.157+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:55.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:55.721+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:55.721+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:55.721+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:55.562+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:55.562+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:55.562+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:56.585 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:56.178+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:56.585 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:56.178+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:56.585 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:56.178+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:56.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:56.691+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:56.691+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:56.691+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:56.584+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:56.584+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:56.584+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:57.564 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:57.168+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:57.564 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:57.168+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:57.565 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:57.168+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:57.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:57.564+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:57.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:57.564+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:57.564+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:57.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:57.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:57.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:58.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:58.180+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:58.180+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:58.180+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:58.781 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:58.524+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:58.781 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:58.524+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:58.781 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:58.524+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:58.782 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:58.670+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:58.782 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:58.670+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:58.782 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:58.670+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:59.508 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:59.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:59.508 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:59.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:59.508 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:17:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:17:59.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:17:59.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:59.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:59.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:17:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:17:59.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:17:59.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:59.507+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:59.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:59.507+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:17:59.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:17:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:17:59.507+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:00.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:00.238+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:00.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:00.238+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:00.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:00.238+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:00.635+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:00.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:00.635+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:00.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:00.635+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:00.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:00.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:00.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:01.536 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:01.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:01.536 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:01.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:01.536 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:01.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:01.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:01.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:01.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:01.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:01.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:01.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:01.852 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:01.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:01.852 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:01.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:01.852 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:01.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:02.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:02.522+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:02.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:02.522+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:02.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:02.522+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:02.279+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:02.279+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:02.279+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:03.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:02.651+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:03.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:02.651+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:03.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:02.651+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:03.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:03.231+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:03.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:03.231+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:03.231+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:03.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:03.604+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:03.604+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:03.604+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:03.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:03.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:03.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:03.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:03.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:03.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:04.542 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:04.220+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:04.542 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:04.220+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:04.542 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:04.220+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:04.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:04.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:04.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:04.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:04.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:04.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:04.541+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:04.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:04.541+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:04.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:04.541+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:05.572 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:05.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:05.572 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:05.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:05.572 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:05.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:05.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:05.571+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:05.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:05.571+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:05.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:05.571+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:05.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:05.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:05.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:06.566 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:06.202+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:06.566 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:06.202+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:06.566 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:06.202+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:06.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:06.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:06.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:06.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:06.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:06.565+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:06.565+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:06.565+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:07.522 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:07.245+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:07.522 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:07.245+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:07.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:07.245+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:07.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:07.636+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:07.636+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:07.636+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:07.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:07.521+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:07.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:07.521+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:07.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:07.521+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:08.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:08.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:08.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:08.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:08.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:08.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:08.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:08.644+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:08.644+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:08.644+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:08.568+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:08.568+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:08.568+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:09.602 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:09.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:09.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:09.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:09.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:09.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:09.621+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:09.621+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:09.621+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:09.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:09.602+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:09.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:09.602+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:09.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:09.602+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:10.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:10.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:10.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:10.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:10.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:10.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:10.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:10.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:10.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:10.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:10.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:10.629+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:10.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:10.629+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:10.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:10.629+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:11.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:11.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:11.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:11.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:11.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:11.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:11.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:11.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:11.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:11.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:11.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:11.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:11.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:11.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:12.587 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:12.181+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:12.587 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:12.181+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:12.587 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:12.181+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:12.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:12.633+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:12.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:12.633+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:12.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:12.633+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:12.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:12.586+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:12.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:12.586+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:12.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:12.586+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:13.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:13.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:13.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:13.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:13.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:13.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:13.656+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:13.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:13.656+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:13.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:13.656+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:13.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:13.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:13.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:14.519 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:14.206+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:14.520 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:14.206+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:14.520 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:14.206+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:14.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:14.681+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:14.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:14.681+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:14.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:14.681+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:14.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:14.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:14.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:15.526 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:15.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:15.526 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:15.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:15.526 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:15.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:15.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:15.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:15.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:15.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:15.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:15.635+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:15.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:15.635+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:15.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:15.635+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:16.512 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:16.193+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:16.512 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:16.193+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:16.512 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:16.193+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:16.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:16.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:16.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:16.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:16.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:16.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:16.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:16.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:16.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:16.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:17.496 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:17.236+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:17.496 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:17.236+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:17.496 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:17.236+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:17.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:17.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:17.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:17.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:17.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:17.654+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:17.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:17.654+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:17.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:17.654+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:18.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:18.260+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:18.260+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:18.260+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:18.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:18.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:18.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:18.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:18.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:18.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:18.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:18.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:18.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:18.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:18.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:19.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:19.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:19.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:19.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:19.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:19.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:19.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:19.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:19.614+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:19.614+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:19.614+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:20.465 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:20.213+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:20.466 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:20.213+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:20.466 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:20.213+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:20.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:20.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:20.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:20.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:20.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:20.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:20.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:21.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:21.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:21.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:21.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:21.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:21.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:21.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:21.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:22.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:21.655+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:22.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:21.655+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:22.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:21.655+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:22.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:22.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:22.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:22.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:22.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:22.247+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:22.247+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:22.247+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:23.036 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:22.622+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:23.036 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:22.622+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:23.037 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:22.622+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:23.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:18:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:18:23.034+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:18:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:23.247+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:23.247+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:23.247+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:23.615 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:23.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:23.615 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:23.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:23.615 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:23.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:24.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:23.614+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:24.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:23.614+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:24.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:23.614+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:24.594 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:24.400+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:24.594 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:24.400+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:24.594 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:24.400+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:24.595 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:24.227+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:24.595 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:24.227+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:24.595 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:24.227+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:24.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:24.593+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:24.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:24.593+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:24.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:24.593+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:25.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:25.423+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:25.423+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:25.423+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:25.257+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:25.257+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:25.257+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:25.620+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:25.620+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:25.620+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:26.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:26.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:26.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:26.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:26.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:26.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:27.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:26.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:26.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:26.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:27.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:27.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:27.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:27.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:27.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:27.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:28.037 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:27.706+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:28.037 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:27.706+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:28.037 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:27.706+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:28.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:18:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:18:28.036+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:18:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:28.233+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:28.233+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:28.233+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:28.691 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:28.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:28.692 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:28.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:28.692 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:28.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:28.691+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:28.691+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:28.691+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:29.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:29.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:29.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:29.195+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:29.195+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:29.195+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:30.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:29.676+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:29.676+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:29.676+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:30.288+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:30.289+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:30.289+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:30.216+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:30.216+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:30.216+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:31.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:30.642+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:30.642+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:30.642+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:31.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:31.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:31.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:31.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:31.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:31.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:32.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:31.643+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:31.643+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:31.643+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:32.572 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:32.334+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:32.572 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:32.334+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:32.572 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:32.334+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:32.572 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:32.280+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:32.572 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:32.280+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:32.572 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:32.280+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:32.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:32.687+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:32.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:32.687+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:32.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:32.687+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:33.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:18:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:18:33.036+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:18:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:33.268+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:33.268+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:33.268+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:33.672 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:33.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:33.672 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:33.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:33.672 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:33.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:33.671+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:33.671+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:33.671+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:34.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:34.227+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:34.227+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:34.227+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:34.360+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:34.360+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:34.360+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:35.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:34.688+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:35.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:34.688+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:34.688+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:35.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:35.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:35.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:35.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:35.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:35.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:35.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:36.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:35.677+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:36.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:35.677+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:36.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:35.677+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:36.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:36.416+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:36.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:36.416+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:36.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:36.416+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:36.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:36.198+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:36.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:36.198+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:36.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:36.198+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:37.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:36.708+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:37.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:36.708+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:37.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:36.708+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:37.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:37.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:37.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:37.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:37.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:37.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:37.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:37.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:38.038 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:37.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:38.038 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:37.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:38.038 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:37.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:38.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:18:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:18:38.037+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 36 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:18:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:38.225+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:38.225+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:38.225+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:38.771 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:38.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:38.771 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:38.496+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:38.771 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:38.496+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:39.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:38.770+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:39.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:38.770+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:39.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:38.770+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:39.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:39.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:39.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:39.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:39.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:39.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:39.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:39.249+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:39.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:39.249+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:39.609 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:39.249+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:39.741+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:39.741+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:39.741+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:40.498 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:40.231+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:40.499 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:40.231+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:40.499 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:40.231+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:40.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:40.715+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:40.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:40.715+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:40.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:40.715+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:40.497+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:40.497+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:40.497+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:41.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:41.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:41.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:41.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:41.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:41.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:41.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:42.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:41.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:42.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:41.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:42.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:41.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:42.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:42.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:42.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:42.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:42.292+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:42.292+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:42.292+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:43.038 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:42.667+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:43.039 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:42.667+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:43.039 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:42.667+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:43.341 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:18:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:18:43.037+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 62 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:18:43.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:43.497+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:43.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:43.497+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:43.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:43.497+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:43.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:43.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:43.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:43.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:43.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:43.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:44.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:43.622+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:44.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:43.622+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:44.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:43.622+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:44.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:44.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:44.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:44.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:44.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:44.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:44.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:44.860 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:44.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:44.860 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:44.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:44.860 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:44.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:45.585 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:45.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:45.585 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:45.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:45.585 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:45.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:45.585 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:45.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:45.586 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:45.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:45.586 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:45.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:45.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:45.584+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:45.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:45.584+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:45.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:45.584+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:46.543 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:46.248+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:46.544 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:46.248+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:46.544 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:46.248+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:46.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:46.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:46.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:46.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:46.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:46.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:46.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:47.581 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:47.297+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:47.581 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:47.298+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:47.581 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:47.298+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:47.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:47.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:47.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:47.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:47.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:47.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:47.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:47.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:47.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:47.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:48.335 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:18:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:18:48.038+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 87 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:18:48.587 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:48.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:48.587 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:48.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:48.588 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:48.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:48.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:48.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:48.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:48.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:48.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:48.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:48.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:48.586+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:48.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:48.586+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:48.586+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:49.578 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:49.288+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:49.578 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:49.288+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:49.578 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:49.288+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:49.578 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:49.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:49.578 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:49.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:49.578 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:49.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:49.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:49.577+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:49.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:49.577+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:49.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:49.577+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:50.501 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:50.240+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:50.501 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:50.240+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:50.501 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:50.240+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:50.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:50.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:50.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:50.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:50.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:50.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:50.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:50.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:50.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:51.527 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:51.255+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:51.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:51.255+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:51.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:51.255+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:51.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:51.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:51.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:51.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:51.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:51.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:51.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:52.576 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:52.264+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:52.576 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:52.264+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:52.576 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:52.264+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:52.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:52.575+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:52.575+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:52.575+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:52.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:52.612+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:52.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:52.612+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:52.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:52.612+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:53.289 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:18:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:18:53.038+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 116 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:18:53.572 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:53.288+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:53.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:53.288+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:53.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:53.288+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:53.619+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:53.619+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:53.619+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:53.571+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:53.571+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:53.571+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:54.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:54.289+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:54.289+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:54.289+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:54.659+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:54.659+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:54.659+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:54.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:54.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:54.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:55.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:55.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:55.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:55.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:55.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:55.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:56.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:55.672+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:55.672+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:55.672+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:56.628 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:56.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:56.628 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:56.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:56.628 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:56.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:56.628 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:56.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:56.629 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:56.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:56.629 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:56.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:56.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:56.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:56.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:57.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:57.388+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:57.388+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:57.388+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:57.634+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:57.634+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:57.634+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:57.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:57.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:57.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:58.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:18:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:18:58.039+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 148 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:18:58.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:58.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:58.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:58.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:58.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:58.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:58.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:58.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:58.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:58.617+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:58.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:58.617+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:58.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:58.617+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:18:59.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:59.540+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:59.540+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:18:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:18:59.540+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:18:59.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:59.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:59.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:59.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:18:59.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:18:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:18:59.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:00.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:59.623+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:00.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:59.623+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:00.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:18:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:18:59.623+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:00.552 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:00.300+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:00.552 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:00.300+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:00.552 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:00.300+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:00.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:00.551+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:00.551+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:00.551+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:00.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:00.583+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:00.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:00.583+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:00.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:00.583+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:01.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:01.332+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:01.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:01.332+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:01.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:01.332+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:01.596 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:01.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:01.596 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:01.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:01.596 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:01.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:01.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:01.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:01.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:01.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:02.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:02.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:02.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:02.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:02.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:03.040 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:02.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:03.040 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:02.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:03.040 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:02.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:03.041 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:02.608+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:03.041 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:02.608+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:03.041 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:02.608+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:03.293 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:03.039+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 177 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:03.293 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:03.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:03.293 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:03.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:03.293 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:03.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:04.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:03.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:04.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:03.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:04.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:03.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:04.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:03.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:04.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:03.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:04.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:03.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:04.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:04.296+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:04.296+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:04.296+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:05.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:04.631+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:04.631+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:04.631+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:04.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:04.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:04.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:05.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:05.324+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:05.324+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:05.324+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:05.571+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:05.571+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:05.571+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:06.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:05.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:05.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:05.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:06.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:06.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:06.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:06.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:06.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:06.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:06.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:06.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:06.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:06.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:06.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:06.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:06.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:07.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:07.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:07.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:07.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:07.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:07.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:07.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:07.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:07.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:07.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:07.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:07.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:07.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:08.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:08.039+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 203 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:08.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:08.572+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:08.572+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:08.572+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:08.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:08.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:08.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:08.389+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:08.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:08.389+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:08.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:08.389+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:09.654 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:09.551+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:09.655 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:09.551+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:09.655 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:09.551+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:09.655 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:09.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:09.655 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:09.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:09.655 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:09.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:10.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:09.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:10.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:09.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:10.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:09.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:10.691 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:10.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:10.691 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:10.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:10.691 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:10.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:10.691 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:10.577+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:10.691 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:10.577+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:10.691 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:10.578+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:11.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:10.690+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:11.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:10.690+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:11.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:10.690+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:11.632 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:11.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:11.633 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:11.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:11.633 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:11.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:11.633 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:11.362+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:11.633 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:11.362+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:11.633 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:11.362+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:12.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:11.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:12.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:11.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:12.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:11.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:12.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:12.347+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:12.347+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:12.347+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:12.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:12.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:12.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:12.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:12.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:12.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:13.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:12.686+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:13.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:12.686+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:13.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:12.686+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:13.343 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:13.040+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 232 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:13.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:13.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:13.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:13.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:13.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:13.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:13.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:13.342+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:13.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:13.342+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:13.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:13.342+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:14.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:13.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:14.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:13.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:14.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:13.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:14.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:14.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:14.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:14.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:14.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:14.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:14.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:14.557+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:14.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:14.557+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:14.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:14.557+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:15.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:15.603+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:15.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:15.603+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:15.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:15.603+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:15.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:15.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:15.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:15.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:15.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:15.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:15.434+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:15.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:15.434+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:15.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:15.434+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:16.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:16.601+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:16.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:16.601+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:16.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:16.601+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:16.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:16.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:16.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:16.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:16.405+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:16.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:16.405+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:16.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:16.405+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:17.646 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:17.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:17.646 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:17.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:17.646 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:17.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:17.646 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:17.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:17.646 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:17.385+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:17.646 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:17.385+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:18.041 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:17.645+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:18.042 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:17.645+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:18.042 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:17.645+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:18.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:18.040+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 260 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:18.653 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:18.601+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:18.653 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:18.601+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:18.653 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:18.601+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:18.654 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:18.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:18.654 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:18.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:18.654 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:18.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:19.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:18.666+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:18.666+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:18.666+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:19.640 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:19.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:19.640 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:19.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:19.640 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:19.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:19.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:19.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:19.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:19.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:19.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:19.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:20.654 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:20.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:20.655 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:20.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:20.655 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:20.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:21.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:20.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:21.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:20.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:21.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:20.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:20.653+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:20.653+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:20.653+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:21.633 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:21.382+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:21.633 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:21.382+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:21.633 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:21.382+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:21.633 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:21.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:21.633 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:21.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:21.634 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:21.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:21.669+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:21.669+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:21.669+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:22.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:22.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:22.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:22.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:22.400+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:22.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:22.400+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:22.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:22.400+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:22.585+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:22.585+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:22.585+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:23.063 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:23.041+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 291 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:23.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:23.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:23.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:23.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:23.445+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:23.445+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:23.445+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:23.630+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:23.630+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:23.630+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:24.796 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:24.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:24.796 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:24.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:24.796 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:24.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:24.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:24.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:24.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:24.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:24.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:24.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:24.797 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:24.589+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:24.797 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:24.589+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:24.797 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:24.589+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:25.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:25.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:25.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:25.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:25.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:25.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:25.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:25.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:25.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:25.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:25.554+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:25.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:25.554+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:25.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:25.554+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:26.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:26.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:26.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:26.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:26.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:26.579+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:26.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:26.579+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:26.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:26.579+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:26.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:26.431+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:26.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:26.431+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:26.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:26.431+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:27.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:27.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:27.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:27.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:27.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:27.446+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:27.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:27.446+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:27.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:27.446+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:27.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:27.582+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:27.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:27.582+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:27.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:27.582+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:28.164 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:28.041+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 317 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:28.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:28.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:28.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:28.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:28.592+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:28.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:28.592+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:28.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:28.592+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:28.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:28.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:28.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:28.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:28.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:28.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:29.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:29.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:29.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:29.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:29.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:29.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:29.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:29.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:29.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:29.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:29.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:29.596+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:29.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:29.596+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:29.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:29.596+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:30.634 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:30.595+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:30.634 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:30.595+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:30.634 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:30.595+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:30.634 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:30.368+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:30.635 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:30.368+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:30.635 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:30.368+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:31.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:30.633+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:31.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:30.633+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:31.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:30.633+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:31.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:31.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:31.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:31.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:31.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:31.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:31.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:31.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:31.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:31.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:31.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:31.545+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:31.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:31.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:31.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:31.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:32.668 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:32.364+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:32.668 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:32.364+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:32.668 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:32.364+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:32.668 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:32.504+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:32.668 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:32.504+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:32.669 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:32.504+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:33.043 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:32.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:33.043 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:32.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:33.043 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:32.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:33.043 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:32.668+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:33.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:33.042+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 350 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:33.715 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:33.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:33.715 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:33.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:33.715 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:33.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:33.716 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:33.384+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:33.716 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:33.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:33.716 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:33.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:33.716 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:33.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:33.716 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:33.547+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:34.013 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:33.714+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:34.013 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:33.714+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:34.013 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:33.714+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:34.013 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:33.715+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:34.720 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:34.357+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:34.720 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:34.357+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:34.720 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:34.357+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:34.720 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:34.358+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:34.720 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:34.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:34.721 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:34.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:34.721 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:34.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:34.721 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:34.536+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:35.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:34.719+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:35.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:34.719+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:35.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:34.719+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:35.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:34.720+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:35.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:35.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:35.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:35.337+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:35.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:35.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:35.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:35.561+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:36.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:35.745+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:36.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:35.745+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:36.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:35.746+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:36.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:35.747+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:36.553 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:36.295+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:36.553 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:36.295+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:36.553 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:36.295+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:36.553 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:36.296+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:36.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:36.782+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:36.782+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:36.782+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:36.783+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:36.552+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:36.552+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:36.552+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:36.553+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:37.540 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:37.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:37.540 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:37.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:37.540 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:37.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:37.540 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:37.271+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:37.751+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:37.751+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:37.751+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:37.752+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:37.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:37.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:37.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:37.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:37.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:37.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:37.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:37.540+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:38.319 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:38.042+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 383 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:38.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:38.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:38.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:38.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:38.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:38.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:38.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:38.319+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:38.542+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:38.542+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:38.542+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:38.543+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:39.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:38.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:39.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:38.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:39.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:38.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:39.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:38.782+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:39.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:39.312+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:39.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:39.312+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:39.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:39.312+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:39.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:39.312+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:39.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:39.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:39.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:39.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:39.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:39.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:39.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:39.529+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:40.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:39.819+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:40.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:39.819+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:40.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:39.819+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:40.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:39.820+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:40.559 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:40.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:40.559 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:40.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:40.559 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:40.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:40.559 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:40.307+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:40.775+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:40.775+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:40.775+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:40.776+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:40.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:40.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:40.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:40.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:40.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:40.559+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:41.298+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:41.298+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:41.298+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:41.299+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:41.517+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:41.517+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:41.517+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:41.517+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:42.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:41.782+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:42.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:41.782+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:42.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:41.782+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:42.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:41.783+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:42.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:42.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:42.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:42.341+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:42.503+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:42.503+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:42.503+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:42.504+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:43.054 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:43.042+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 412 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:43.054 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:42.811+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:43.054 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:42.811+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:43.054 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:42.811+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:43.054 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:42.812+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:43.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:43.345+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:43.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:43.345+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:43.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:43.345+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:43.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:43.346+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:43.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:43.483+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:43.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:43.483+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:43.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:43.483+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:43.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:43.484+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:44.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:43.841+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:44.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:43.841+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:44.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:43.841+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:44.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:43.842+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:44.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:44.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:44.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:44.319+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:44.500+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:44.500+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:44.500+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:44.501+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:44.813+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:44.813+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:44.813+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:44.814+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:45.831 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:45.360+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:45.831 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:45.360+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:45.831 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:45.360+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:45.831 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:45.361+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:45.831 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:45.518+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:45.831 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:45.518+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:45.831 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:45.518+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:45.831 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:45.519+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:46.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:45.829+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:45.829+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:45.829+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:45.830+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:46.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:46.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:46.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:46.547+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:46.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:46.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:46.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:46.392+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:47.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:46.820+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:47.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:46.820+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:47.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:46.820+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:47.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:46.821+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:47.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:47.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:47.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:47.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:47.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:47.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:47.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:47.500+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:47.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:47.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:47.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:47.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:47.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:47.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:47.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:47.370+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:48.166 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:48.043+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 438 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:48.167 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:47.869+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:48.167 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:47.869+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:48.167 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:47.869+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:48.167 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:47.870+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:48.516 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:48.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:48.517 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:48.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:48.517 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:48.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:48.517 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:48.372+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:48.849 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:48.515+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:48.850 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:48.515+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:48.850 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:48.515+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:48.850 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:48.516+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:49.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:48.847+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:49.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:48.847+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:48.847+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:48.848+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:49.504+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:49.504+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:49.504+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:49.505+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:49.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:49.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:49.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:49.328+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:50.339 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:49.872+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:50.339 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:49.872+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:50.340 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:49.872+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:50.340 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:49.873+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:50.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:50.455+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:50.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:50.455+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:50.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:50.455+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:50.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:50.456+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:50.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:50.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:50.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:50.338+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:51.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:50.840+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:50.840+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:50.840+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:50.841+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:51.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:51.445+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:51.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:51.445+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:51.608 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:51.445+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:51.608 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:51.445+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:51.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:51.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:51.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:51.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:51.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:51.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:51.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:51.337+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:52.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:51.831+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:51.831+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:51.831+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:51.832+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:52.802 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:52.381+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:52.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:52.381+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:52.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:52.381+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:52.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:52.381+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:52.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:52.425+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:52.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:52.425+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:52.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:52.425+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:52.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:52.426+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:53.055 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:53.042+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 472 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:53.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:52.853+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:53.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:52.853+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:53.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:52.853+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:53.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:52.854+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:53.382+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:53.382+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:53.382+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:53.383+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:53.431+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:53.431+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:53.431+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:53.432+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:54.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:53.893+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:54.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:53.893+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:54.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:53.894+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:54.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:53.895+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:54.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:54.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:54.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:54.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:54.404+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:54.417+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:54.417+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:54.417+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:54.418+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:55.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:54.858+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:55.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:54.858+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:55.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:54.858+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:55.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:54.859+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:55.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:55.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:55.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:55.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:55.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:55.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:55.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:55.359+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:55.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:55.460+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:55.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:55.460+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:55.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:55.460+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:55.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:55.460+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:56.336 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:55.874+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:56.337 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:55.874+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:56.337 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:55.874+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:56.337 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:55.874+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:56.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:56.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:56.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:56.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:56.424+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:56.335+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:56.335+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:56.335+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:56.336+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:57.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:56.874+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:57.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:56.874+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:57.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:56.874+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:57.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:56.874+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:57.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:57.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:57.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:57.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:57.416+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:57.381+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:57.381+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:57.381+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:57.382+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:58.168 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:19:58.044+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 504 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:19:58.168 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:57.886+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:58.168 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:57.886+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:58.168 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:57.886+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:58.168 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:57.886+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:58.424 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:58.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:58.424 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:58.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:58.424 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:58.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:58.424 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:58.364+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:58.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:58.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:58.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:58.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:58.424+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:59.319 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:58.873+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:59.320 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:58.873+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:59.320 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:58.873+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:19:59.320 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:58.874+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:59.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:59.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:59.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:59.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:59.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:59.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:19:59.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:19:59.319+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:19:59.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:59.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:59.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:59.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:59.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:59.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:19:59.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:19:59.380+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:00.341 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:59.862+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:00.342 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:59.862+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:00.342 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:59.862+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:00.342 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:19:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:19:59.862+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:00.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:00.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:00.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:00.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:00.341+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:00.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:00.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:00.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:00.354+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:01.303 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:00.880+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:01.303 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:00.881+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:01.303 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:00.881+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:01.303 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:00.882+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:01.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:01.302+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:01.302+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:01.302+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:01.303+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:01.344+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:01.344+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:01.344+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:01.345+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:02.271 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:01.889+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:02.271 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:01.889+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:02.272 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:01.889+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:02.272 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:01.889+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:02.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:02.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:02.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:02.271+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:02.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:02.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:02.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:02.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:02.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:02.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:02.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:02.367+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:03.055 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:03.044+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 533 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:03.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:02.880+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:03.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:02.880+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:03.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:02.880+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:03.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:02.881+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:03.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:03.243+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:03.243+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:03.243+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:03.244+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:03.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:03.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:03.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:03.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:03.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:03.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:03.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:03.405+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:04.199 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:03.856+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:04.199 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:03.856+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:04.199 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:03.856+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:04.200 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:03.857+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:04.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:04.198+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:04.198+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:04.198+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:04.199+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:04.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:04.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:04.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:04.400+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:04.811+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:04.811+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:04.811+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:04.812+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:05.371 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:05.150+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:05.371 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:05.150+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:05.371 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:05.150+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:05.371 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:05.151+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:05.798 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:05.370+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:05.798 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:05.370+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:05.798 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:05.370+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:05.798 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:05.370+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:06.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:05.797+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:06.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:05.797+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:06.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:05.797+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:06.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:05.797+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:06.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:06.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:06.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:06.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:06.107+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:06.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:06.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:06.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:06.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:06.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:06.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:06.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:06.346+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:07.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:06.770+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:07.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:06.770+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:07.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:06.770+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:07.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:06.770+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:07.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:07.317+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:07.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:07.317+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:07.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:07.317+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:07.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:07.318+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:07.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:07.144+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:07.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:07.145+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:07.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:07.145+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:07.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:07.145+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:08.046 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:07.774+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:08.046 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:07.774+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:08.046 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:07.774+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:08.046 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:07.775+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:08.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:08.045+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 559 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:08.275+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:08.275+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:08.275+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:08.275+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:08.120+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:08.120+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:08.120+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:08.120+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:09.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:08.733+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:09.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:08.733+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:09.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:08.733+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:09.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:08.734+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:09.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:09.135+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:09.135+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:09.135+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:09.136+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:09.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:09.240+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:09.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:09.240+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:09.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:09.240+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:09.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:09.240+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:10.089 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:09.705+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:10.089 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:09.705+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:10.089 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:09.705+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:10.089 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:09.706+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:10.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:10.219+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:10.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:10.219+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:10.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:10.219+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:10.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:10.220+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:10.088+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:10.088+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:10.088+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:10.089+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:11.080 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:10.746+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:11.080 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:10.746+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:11.080 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:10.746+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:11.080 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:10.747+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:11.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:11.079+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:11.079+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:11.079+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:11.080+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:11.187+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:11.187+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:11.187+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:11.188+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:12.097 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:11.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:12.097 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:11.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:12.097 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:11.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:12.097 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:11.781+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:12.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:12.171+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:12.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:12.171+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:12.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:12.171+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:12.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:12.172+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:12.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:12.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:12.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:12.097+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:13.055 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:13.045+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 587 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:13.055 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:12.804+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:13.055 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:12.804+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:13.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:12.804+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:13.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:12.805+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:13.102+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:13.102+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:13.102+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:13.103+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:13.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:13.152+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:13.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:13.152+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:13.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:13.152+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:13.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:13.152+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:13.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:13.771+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:13.771+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:13.771+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:13.772+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:14.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:14.131+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:14.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:14.131+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:14.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:14.131+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:14.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:14.132+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:14.117+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:14.117+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:14.117+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:14.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:14.117+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:15.093 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:14.779+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:15.093 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:14.779+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:15.093 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:14.779+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:15.094 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:14.780+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:15.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:15.092+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:15.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:15.092+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:15.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:15.092+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:15.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:15.093+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:15.093+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:15.093+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:15.093+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:15.094+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:16.048 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:15.788+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:16.048 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:15.788+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:16.048 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:15.788+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:16.049 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:15.789+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:16.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:16.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:16.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:16.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:16.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:16.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:16.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:16.095+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:16.047+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:16.047+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:16.047+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:16.048+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:17.064 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:16.786+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:17.064 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:16.786+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:17.064 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:16.786+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:17.064 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:16.787+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:17.063+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:17.063+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:17.063+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:17.064+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:17.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:17.115+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:17.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:17.115+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:17.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:17.115+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:17.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:17.116+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:18.047 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:17.751+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:18.047 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:17.751+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:18.047 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:17.751+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:18.047 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:17.751+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:18.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:18.045+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 616 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:18.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:18.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:18.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:18.094+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:18.089+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:18.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:18.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:18.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:18.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:18.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:18.090+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:19.052 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:19.050+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:19.052 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:19.050+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:19.052 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:19.050+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:19.053 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:18.800+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:19.053 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:18.800+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:19.053 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:18.800+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:19.053 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:18.801+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:19.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:19.052+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:19.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:19.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:19.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:19.097+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:20.086 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:19.792+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:20.086 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:19.792+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:20.086 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:19.792+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:20.086 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:19.793+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:20.086 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:20.024+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:20.087 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:20.024+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:20.087 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:20.024+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:20.087 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:20.025+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:20.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:20.085+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:20.085+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:20.085+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:20.086+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:21.034 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:20.754+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:21.034 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:20.754+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:21.034 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:20.754+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:21.034 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:20.755+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:21.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:21.130+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:21.130+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:21.130+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:21.131+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:21.033+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:21.033+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:21.033+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:21.034+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:21.994 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:21.708+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:21.995 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:21.708+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:21.995 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:21.708+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:21.995 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:21.708+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:21.993+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:21.993+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:21.993+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:21.994+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:22.167+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:22.167+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:22.167+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:22.168+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:23.017 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:22.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:23.017 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:22.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:23.017 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:22.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:23.017 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:22.664+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:23.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:23.046+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 644 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:23.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:23.015+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:23.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:23.015+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:23.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:23.015+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:23.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:23.016+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:23.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:23.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:23.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:23.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:23.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:23.192+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:23.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:23.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:23.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:23.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:23.677+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:24.038+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:24.038+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:24.038+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:24.039+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:24.171+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:24.171+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:24.171+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:24.172+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:25.033 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:24.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:25.033 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:24.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:25.033 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:24.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:25.033 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:24.651+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:25.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:25.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:25.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:25.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:25.033+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:25.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:25.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:25.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:25.193+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:26.071 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:25.633+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:26.071 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:25.633+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:26.071 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:25.633+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:26.071 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:25.634+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:26.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:26.189+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:26.189+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:26.189+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:26.189+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:26.070+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:26.070+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:26.070+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:26.071+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:27.060 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:26.668+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:27.060 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:26.668+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:27.060 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:26.668+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:27.061 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:26.669+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:27.195+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:27.195+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:27.195+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:27.196+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:27.059+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:27.059+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:27.059+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:27.060+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:28.028 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:27.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:28.028 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:27.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:28.028 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:27.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:28.028 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:27.689+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:28.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:28.046+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 671 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:28.152+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:28.152+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:28.152+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:28.153+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:28.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:28.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:28.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:28.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:28.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:28.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:28.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:28.028+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:29.043 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:28.685+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:29.043 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:28.685+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:29.043 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:28.685+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:29.043 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:28.686+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:29.308 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:29.105+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:29.309 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:29.105+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:29.309 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:29.105+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:29.309 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:29.106+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:29.309 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:29.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:29.309 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:29.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:29.309 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:29.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:29.309 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:29.043+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:30.070 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:29.658+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:30.071 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:29.658+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:30.071 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:29.658+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:30.071 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:29.659+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:30.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:30.069+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:30.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:30.069+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:30.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:30.069+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:30.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:30.070+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:30.118+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:30.118+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:30.118+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:30.118+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:31.101 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:30.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:31.101 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:30.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:31.101 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:30.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:31.101 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:30.640+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:31.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:31.108+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:31.108+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:31.108+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:31.108+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:31.100+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:31.100+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:31.100+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:31.101+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:32.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:31.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:32.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:31.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:32.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:31.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:32.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:31.612+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:32.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:32.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:32.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:32.104+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:32.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:32.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:32.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:32.128+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:32.588+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:32.588+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:32.588+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:32.589+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:33.058 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:33.046+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 699 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:33.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:33.135+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:33.135+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:33.135+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:33.136+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:33.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:33.140+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:33.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:33.140+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:33.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:33.140+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:33.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:33.141+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:33.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:33.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:33.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:33.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:33.581+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:34.332 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:34.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:34.332 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:34.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:34.332 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:34.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:34.332 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:34.104+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:34.332 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:34.121+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:34.332 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:34.121+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:34.332 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:34.121+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:34.332 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:34.122+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:34.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:34.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:34.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:34.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:34.573+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:35.452 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:35.147+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:35.452 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:35.147+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:35.452 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:35.147+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:35.452 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:35.147+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:35.452 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:35.161+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:35.452 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:35.161+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:35.452 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:35.161+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:35.453 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:35.161+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:35.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:35.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:35.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:35.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:35.560+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:36.582 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:36.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:36.582 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:36.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:36.582 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:36.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:36.583 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:36.113+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:36.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:36.169+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:36.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:36.169+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:36.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:36.169+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:36.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:36.170+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:36.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:36.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:36.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:36.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:36.581+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:37.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:37.162+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:37.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:37.162+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:37.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:37.163+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:37.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:37.163+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:37.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:37.139+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:37.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:37.139+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:37.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:37.139+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:37.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:37.139+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:37.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:37.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:37.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:37.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:37.573+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:38.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:38.047+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 731 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:38.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:38.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:38.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:38.167+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:38.098+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:38.098+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:38.098+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:38.100+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:38.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:38.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:38.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:38.530+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:39.144+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:39.144+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:39.144+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:39.145+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:39.100+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:39.100+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:39.100+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:39.101+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:39.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:39.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:39.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:39.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:39.547+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:40.542 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:40.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:40.542 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:40.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:40.542 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:40.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:40.542 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:40.156+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:40.543 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:40.147+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:40.543 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:40.147+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:40.543 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:40.147+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:40.543 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:40.147+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:40.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:40.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:40.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:40.542+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:41.166+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:41.166+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:41.166+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:41.167+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:41.205+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:41.205+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:41.205+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:41.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:41.206+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:41.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:41.538+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:41.538+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:41.538+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:41.539+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:42.505 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:42.196+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:42.505 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:42.196+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:42.505 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:42.196+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:42.505 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:42.197+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:42.505 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:42.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:42.505 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:42.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:42.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:42.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:42.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:42.184+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:42.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:42.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:42.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:42.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:42.505+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:43.060 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:43.048+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 762 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:43.151+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:43.151+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:43.151+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:43.151+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:43.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:43.220+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:43.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:43.220+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:43.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:43.220+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:43.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:43.221+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:43.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:43.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:43.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:43.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:43.483+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:44.468 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:44.159+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:44.468 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:44.159+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:44.468 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:44.159+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:44.468 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:44.160+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:44.468 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:44.206+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:44.469 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:44.206+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:44.469 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:44.206+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:44.469 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:44.206+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:44.753 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:44.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:44.753 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:44.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:44.753 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:44.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:44.753 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:44.468+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:45.419 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:45.241+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:45.420 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:45.241+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:45.420 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:45.241+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:45.420 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:45.242+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:45.420 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:45.137+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:45.420 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:45.137+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:45.420 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:45.137+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:45.420 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:45.138+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:45.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:45.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:45.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:45.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:45.419+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:46.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:46.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:46.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:46.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:46.095+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:46.271+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:46.271+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:46.271+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:46.271+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:46.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:46.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:46.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:46.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:46.440+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:47.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:47.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:47.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:47.057+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:47.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:47.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:47.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:47.304+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:47.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:47.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:47.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:47.399+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:48.313 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:48.049+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 789 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:48.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:48.102+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:48.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:48.102+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:48.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:48.102+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:48.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:48.103+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:48.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:48.312+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:48.312+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:48.312+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:48.313+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:48.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:48.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:48.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:48.411+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:49.461 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:49.275+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:49.461 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:49.275+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:49.461 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:49.275+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:49.462 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:49.276+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:49.462 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:49.120+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:49.462 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:49.120+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:49.462 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:49.120+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:49.462 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:49.121+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:49.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:49.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:49.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:49.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:49.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:49.461+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:50.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:50.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:50.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:50.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:50.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:50.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:50.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:50.319+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:50.075+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:50.075+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:50.075+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:50.075+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:50.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:50.479+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:50.479+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:50.479+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:50.480+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:51.515 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:51.354+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:51.515 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:51.354+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:51.515 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:51.354+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:51.515 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:51.355+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:51.515 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:51.118+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:51.515 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:51.118+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:51.515 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:51.118+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:51.515 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:51.119+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:51.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:51.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:51.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:51.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:51.514+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:52.486 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:52.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:52.486 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:52.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:52.486 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:52.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:52.487 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:52.306+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:52.487 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:52.168+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:52.487 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:52.168+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:52.487 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:52.168+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:52.487 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:52.169+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:52.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:52.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:52.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:52.486+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:53.058 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:53.049+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 817 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:53.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:53.200+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:53.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:53.200+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:53.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:53.200+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:53.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:53.201+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:53.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:53.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:53.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:53.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:53.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:53.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:53.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:53.522+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:53.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:53.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:53.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:53.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:53.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:53.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:53.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:53.330+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:54.505 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:54.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:54.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:54.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:54.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:54.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:54.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:54.294+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:54.506 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:54.201+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:54.506 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:54.201+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:54.506 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:54.201+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:54.506 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:54.201+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:54.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:54.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:54.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:54.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:54.505+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:55.484 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:55.256+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:55.485 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:55.256+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:55.485 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:55.256+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:55.485 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:55.257+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:55.485 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:55.189+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:55.485 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:55.189+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:55.485 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:55.189+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:55.485 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:55.190+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:55.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:55.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:55.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:55.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:55.484+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:56.502 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:56.241+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:56.503 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:56.241+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:56.503 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:56.241+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:56.503 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:56.242+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:56.503 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:56.185+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:56.503 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:56.185+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:56.503 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:56.185+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:56.503 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:56.186+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:56.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:56.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:56.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:56.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:56.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:56.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:56.502+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:57.473 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:57.211+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:57.473 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:57.211+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:57.473 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:57.211+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:57.474 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:57.211+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:57.474 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:57.180+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:57.474 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:57.180+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:57.474 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:57.180+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:57.474 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:57.180+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:57.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:57.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:57.929 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:57.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:57.929 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:57.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:57.929 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:57.473+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:58.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:20:58.049+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 846 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:20:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:58.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:58.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:58.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:58.198+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:58.175+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:58.175+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:58.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:58.175+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:58.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:58.176+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:58.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:58.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:58.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:58.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:58.516+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:59.516 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:59.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:59.517 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:59.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:59.517 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:59.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:20:59.517 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:20:59.215+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:59.517 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:59.176+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:59.517 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:59.176+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:59.517 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:59.176+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:20:59.517 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:20:59.177+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:20:59.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:59.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:59.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:59.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:20:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:20:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:20:59.516+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:00.567 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:00.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:00.567 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:00.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:00.567 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:00.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:00.567 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:00.218+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:00.567 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:00.189+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:00.567 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:00.189+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:00.568 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:00.189+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:00.568 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:00.189+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:00.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:00.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:00.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:00.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:00.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:00.566+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:01.230+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:01.230+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:01.230+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:01.230+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:01.170+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:01.170+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:01.170+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:01.170+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:02.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:01.610+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:02.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:01.610+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:02.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:01.610+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:02.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:01.610+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:02.233+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:02.233+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:02.233+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:02.234+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:02.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:02.163+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:02.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:02.163+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:02.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:02.163+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:02.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:02.164+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:03.051 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:02.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:03.051 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:02.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:03.051 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:02.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:03.051 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:02.659+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:03.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:03.050+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 874 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:03.126+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:03.126+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:03.126+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:03.127+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:03.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:03.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:03.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:03.204+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:04.094 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:03.643+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:04.094 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:03.643+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:04.094 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:03.643+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:04.095 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:03.644+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:04.203+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:04.203+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:04.203+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:04.204+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:04.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:04.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:04.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:04.094+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:04.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:04.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:04.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:04.614+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:05.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:05.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:05.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:05.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:05.219+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:05.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:05.131+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:05.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:05.131+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:05.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:05.131+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:05.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:05.132+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:06.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:05.651+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:06.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:05.651+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:06.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:05.651+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:06.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:05.652+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:06.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:06.148+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:06.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:06.148+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:06.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:06.148+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:06.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:06.149+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:06.240+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:06.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:06.240+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:06.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:06.240+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:06.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:06.241+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:07.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:06.604+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:07.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:06.604+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:07.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:06.604+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:07.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:06.605+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:07.568 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:07.193+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:07.568 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:07.193+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:07.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:07.193+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:07.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:07.194+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:07.569 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:07.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:07.569 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:07.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:07.569 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:07.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:07.569 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:07.108+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:07.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:07.567+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:07.567+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:07.567+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:07.568+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:08.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:08.050+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 901 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:08.232+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:08.232+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:08.232+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:08.233+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:08.116+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:08.116+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:08.116+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:08.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:08.117+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:08.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:08.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:08.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:08.582+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:09.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:09.194+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:09.194+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:09.194+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:09.195+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:09.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:09.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:09.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:09.095+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:09.563+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:09.563+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:09.563+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:09.564+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:10.530 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:10.170+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:10.530 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:10.170+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:10.530 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:10.170+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:10.530 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:10.171+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:10.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:10.122+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:10.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:10.122+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:10.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:10.122+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:10.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:10.123+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:10.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:10.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:10.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:10.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:10.529+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:11.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:11.208+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:11.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:11.208+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:11.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:11.208+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:11.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:11.209+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:11.506 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:11.138+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:11.506 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:11.138+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:11.506 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:11.138+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:11.506 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:11.139+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:11.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:11.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:11.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:11.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:11.505+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:12.482 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:12.196+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:12.483 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:12.196+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:12.483 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:12.196+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:12.483 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:12.197+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:12.483 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:12.134+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:12.483 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:12.134+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:12.483 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:12.134+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:12.483 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:12.135+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:12.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:12.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:12.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:12.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:12.482+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:13.062 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:13.051+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 929 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:13.185+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:13.185+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:13.185+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:13.185+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:13.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:13.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:13.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:13.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:13.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:13.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:13.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:13.107+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:13.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:13.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:13.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:13.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:13.507+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:14.489 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:14.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:14.489 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:14.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:14.489 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:14.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:14.489 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:14.220+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:14.490 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:14.109+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:14.490 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:14.109+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:14.490 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:14.109+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:14.490 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:14.110+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:14.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:14.488+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:14.488+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:14.488+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:14.489+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:15.536 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:15.212+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:15.536 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:15.212+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:15.536 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:15.212+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:15.537 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:15.213+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:15.537 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:15.144+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:15.537 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:15.144+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:15.537 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:15.144+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:15.537 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:15.145+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:15.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:15.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:15.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:15.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:15.536+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:16.476 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:16.210+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:16.477 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:16.210+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:16.477 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:16.210+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:16.477 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:16.211+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:16.477 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:16.114+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:16.477 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:16.114+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:16.477 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:16.114+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:16.477 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:16.115+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:16.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:16.552+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:16.552+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:16.552+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:16.553+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:17.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:17.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:17.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:17.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:17.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:17.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:17.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:17.131+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:17.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:17.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:17.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:17.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:17.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:17.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:17.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:17.205+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:17.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:17.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:17.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:17.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:17.520+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:18.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:18.050+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 958 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:18.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:18.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:18.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:18.225+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:18.165+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:18.165+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:18.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:18.165+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:18.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:18.166+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:18.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:18.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:18.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:18.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:18.500+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:19.488 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:19.192+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:19.488 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:19.192+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:19.488 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:19.192+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:19.488 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:19.192+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:19.488 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:19.266+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:19.488 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:19.266+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:19.489 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:19.266+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:19.489 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:19.267+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:19.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:19.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:19.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:19.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:19.487+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:20.459 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:20.275+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:20.460 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:20.275+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:20.460 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:20.275+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:20.460 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:20.276+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:20.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:20.201+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:20.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:20.201+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:20.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:20.201+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:20.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:20.201+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:20.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:20.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:20.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:20.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:20.458+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:21.497 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:21.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:21.497 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:21.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:21.497 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:21.263+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:21.497 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:21.263+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:21.498 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:21.182+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:21.498 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:21.182+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:21.498 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:21.182+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:21.498 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:21.183+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:21.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:21.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:21.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:21.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:21.496+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:22.475 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:22.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:22.475 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:22.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:22.475 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:22.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:22.475 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:22.266+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:22.476 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:22.213+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:22.476 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:22.213+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:22.476 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:22.213+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:22.476 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:22.214+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:22.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:22.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:22.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:22.475+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:23.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:23.052+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 986 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:23.278+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:23.278+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:23.278+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:23.279+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:23.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:23.250+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:23.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:23.250+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:23.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:23.250+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:23.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:23.251+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:23.479+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:23.479+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:23.479+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:23.480+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:24.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:24.298+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:24.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:24.298+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:24.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:24.298+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:24.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:24.299+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:24.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:24.448+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:24.448+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:24.448+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:24.449+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:24.308+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:24.308+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:24.308+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:24.309+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:25.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:25.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:25.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:25.425+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:25.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:25.349+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:25.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:25.349+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:25.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:25.349+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:25.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:25.350+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:25.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:25.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:25.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:25.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:25.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:25.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:25.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:25.329+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:26.334+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:26.334+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:26.334+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:26.335+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:26.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:26.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:26.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:26.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:26.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:26.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:26.452+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:26.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:26.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:26.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:26.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:26.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:26.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:26.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:26.356+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:27.314+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:27.314+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:27.314+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:27.315+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:27.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:27.406+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:27.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:27.406+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:27.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:27.406+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:27.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:27.407+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:27.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:27.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:27.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:27.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:27.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:27.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:27.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:27.340+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:28.333 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:28.053+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1013 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:28.334 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:28.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:28.334 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:28.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:28.334 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:28.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:28.334 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:28.281+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:28.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:28.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:28.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:28.333+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:28.444+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:28.444+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:28.444+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:28.445+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:29.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:29.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:29.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:29.433+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:29.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:29.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:29.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:29.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:29.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:29.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:29.352+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:29.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:29.327+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:29.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:29.327+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:29.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:29.327+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:29.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:29.328+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:30.453+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:30.453+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:30.453+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:30.454+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:30.364+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:30.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:30.364+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:30.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:30.364+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:30.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:30.364+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:30.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:30.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:30.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:30.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:30.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:30.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:30.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:30.346+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:31.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:31.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:31.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:31.439+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:31.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:31.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:31.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:31.363+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:31.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:31.337+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:31.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:31.337+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:31.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:31.337+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:31.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:31.338+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:32.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:32.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:32.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:32.415+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:32.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:32.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:32.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:32.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:32.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:32.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:32.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:32.379+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:32.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:32.367+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:32.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:32.367+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:32.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:32.367+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:32.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:32.367+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:33.327 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:33.053+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1041 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:33.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:33.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:33.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:33.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:33.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:33.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:33.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:33.447+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:33.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:33.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:33.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:33.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:33.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:33.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:33.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:33.393+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:33.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:33.327+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:33.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:33.327+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:33.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:33.327+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:33.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:33.327+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:34.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:34.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:34.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:34.349+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:34.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:34.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:34.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:34.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:34.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:34.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:34.477+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:34.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:34.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:34.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:34.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:34.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:34.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:34.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:34.340+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:35.510+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:35.510+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:35.510+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:35.511+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:35.371+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:35.371+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:35.371+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:35.372+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:35.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:35.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:35.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:35.384+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:36.555+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:36.555+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:36.555+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:36.555+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:36.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:36.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:36.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:36.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:36.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:36.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:36.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:36.425+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:36.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:36.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:36.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:36.346+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:37.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:37.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:37.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:37.590+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:37.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:37.386+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:37.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:37.386+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:37.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:37.386+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:37.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:37.387+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:37.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:37.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:37.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:37.380+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:38.340 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:38.054+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1085 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:38.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:38.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:38.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:38.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:38.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:38.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:38.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:38.371+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:38.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:38.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:38.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:38.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:38.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:38.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:38.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:38.340+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:39.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:38.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:39.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:38.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:39.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:38.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:39.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:38.619+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:39.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:39.308+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:39.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:39.308+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:39.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:39.308+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:39.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:39.309+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:39.589 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:39.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:39.589 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:39.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:39.589 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:39.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:39.589 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:39.349+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:39.588+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:39.588+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:39.588+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:39.589+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:40.314+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:40.314+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:40.314+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:40.314+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:40.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:40.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:40.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:40.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:40.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:40.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:40.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:40.324+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:41.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:40.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:41.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:40.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:41.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:40.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:41.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:40.606+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:41.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:41.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:41.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:41.265+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:41.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:41.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:41.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:41.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:41.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:41.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:41.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:41.347+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:42.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:41.632+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:42.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:41.632+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:42.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:41.632+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:42.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:41.633+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:42.259+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:42.259+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:42.259+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:42.259+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:42.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:42.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:42.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:42.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:42.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:42.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:42.351+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:43.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:42.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:43.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:42.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:43.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:42.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:43.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:42.677+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:43.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:43.054+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1128 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:43.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:43.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:43.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:43.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:43.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:43.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:43.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:43.271+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:43.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:43.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:43.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:43.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:43.628+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:43.396+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:43.396+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:43.396+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:43.397+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:44.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:44.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:44.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:44.266+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:44.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:44.409+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:44.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:44.409+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:44.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:44.409+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:44.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:44.410+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:45.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:44.645+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:44.645+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:44.645+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:44.647+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:45.236+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:45.236+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:45.236+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:45.237+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:45.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:45.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:45.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:45.431+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:45.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:45.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:45.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:45.624+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:46.382 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:46.269+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:46.382 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:46.269+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:46.382 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:46.269+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:46.382 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:46.269+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:46.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:46.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:46.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:46.628+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:46.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:46.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:46.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:46.382+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:47.296+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:47.296+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:47.296+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:47.297+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:47.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:47.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:47.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:47.352+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:48.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:47.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:48.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:47.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:48.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:47.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:48.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:47.635+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:48.313 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:48.055+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1170 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:48.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:48.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:48.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:48.313+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:48.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:48.400+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:48.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:48.400+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:48.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:48.400+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:48.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:48.401+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:48.616+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:48.616+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:48.616+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:48.617+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:49.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:49.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:49.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:49.292+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:49.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:49.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:49.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:49.365+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:49.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:49.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:49.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:49.606+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:50.583 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:50.274+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:50.583 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:50.274+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:50.583 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:50.274+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:50.583 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:50.274+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:50.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:50.325+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:50.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:50.325+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:50.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:50.325+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:50.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:50.326+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:50.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:50.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:50.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:50.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:50.583+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:51.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:51.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:51.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:51.294+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:51.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:51.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:51.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:51.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:51.535+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:51.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:51.360+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:51.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:51.360+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:51.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:51.360+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:51.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:51.360+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:21:51.907+0000 7f34aff5b700 0 monclient(hunting): authenticate timed out after 300 2024-02-10T06:21:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-10T06:21:52.582 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:52.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:52.582 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:52.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:52.582 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:52.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:52.582 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:52.324+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:52.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:52.363+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:52.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:52.363+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:52.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:52.363+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:52.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:52.364+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:52.697 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T06:21:52.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:52.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:52.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:52.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:52.581+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:52.973 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/mon.smithi089/config 2024-02-10T06:21:53.342 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:53.055+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1213 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:53.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:53.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:53.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:53.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:53.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:53.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:53.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:53.583+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:53.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:53.412+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:53.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:53.412+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:53.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:53.412+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:53.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:53.413+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:53.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:53.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:53.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:53.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:53.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:53.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:53.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:53.342+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:54.446 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:54.389+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:54.446 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:54.389+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:54.446 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:54.389+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:54.446 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:54.390+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:54.738 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:54.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:54.738 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:54.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:54.738 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:54.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:54.738 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:54.548+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:54.738 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:54.444+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:54.739 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:54.444+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:54.739 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:54.444+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:54.739 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:54.445+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:55.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:55.436+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:55.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:55.436+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:55.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:55.436+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:55.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:55.437+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:55.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:55.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:55.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:55.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:55.527+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:55.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:55.410+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:55.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:55.410+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:55.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:55.410+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:55.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:55.410+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:56.372+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:56.372+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:56.372+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:56.373+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:56.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:56.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:56.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:56.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:56.570+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:56.418+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:56.418+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:56.418+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:56.419+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:57.366+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:57.366+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:57.366+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:57.367+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:57.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:57.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:57.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:57.526+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:57.432+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:57.432+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:57.432+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:57.433+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:58.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:21:58.056+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1257 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:21:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:58.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:58.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:58.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:58.571+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:58.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:58.399+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:58.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:58.399+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:58.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:58.399+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:58.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:58.400+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:58.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:58.391+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:58.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:58.391+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:58.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:58.391+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:58.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:58.391+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:59.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:59.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:59.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:21:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:21:59.374+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:59.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:59.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:59.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:59.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:21:59.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:21:59.573+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:21:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:59.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:59.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:59.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:21:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:21:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:21:59.415+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:00.590 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:00.465+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:00.590 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:00.465+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:00.590 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:00.465+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:00.590 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:00.466+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:00.590 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:00.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:00.590 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:00.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:00.590 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:00.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:00.590 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:00.335+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:00.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:00.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:00.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:00.589+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:01.599 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:01.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:01.599 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:01.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:01.599 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:01.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:01.599 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:01.328+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:01.599 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:01.455+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:01.599 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:01.455+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:01.599 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:01.455+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:01.599 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:01.455+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:01.596+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:01.596+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:01.596+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:01.597+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:02.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:02.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:02.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:02.349+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:02.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:02.556+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:02.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:02.556+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:02.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:02.556+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:02.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:02.557+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:02.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:02.480+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:02.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:02.480+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:02.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:02.480+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:02.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:02.481+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:03.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:03.056+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1300 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:03.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:03.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:03.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:03.537+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:03.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:03.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:03.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:03.388+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:03.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:03.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:03.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:03.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:03.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:03.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:03.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:03.452+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:04.488+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:04.488+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:04.488+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:04.488+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:04.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:04.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:04.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:04.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:04.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:04.352+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:04.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:04.473+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:04.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:04.473+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:04.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:04.473+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:04.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:04.473+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:05.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:05.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:05.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:05.477+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:05.311+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:05.311+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:05.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:05.311+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:05.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:05.311+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:05.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:05.424+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:05.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:05.424+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:05.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:05.424+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:05.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:05.424+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:06.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:06.273+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:06.273+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:06.273+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:06.274+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:06.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:06.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:06.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:06.439+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:06.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:06.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:06.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:06.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:06.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:06.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:06.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:06.437+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:07.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:07.238+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:07.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:07.238+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:07.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:07.238+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:07.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:07.239+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:07.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:07.484+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:07.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:07.484+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:07.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:07.484+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:07.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:07.485+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:07.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:07.409+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:07.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:07.409+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:07.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:07.409+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:07.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:07.410+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:08.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:08.057+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1343 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:08.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:08.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:08.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:08.234+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:08.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:08.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:08.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:08.515+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:08.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:08.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:08.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:08.362+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:09.482 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:09.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:09.482 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:09.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:09.482 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:09.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:09.482 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:09.205+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:09.482 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:09.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:09.482 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:09.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:09.482 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:09.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:09.483 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:09.334+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:09.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:09.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:09.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:09.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:09.482+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:10.482 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:10.194+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:10.483 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:10.194+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:10.483 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:10.194+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:10.483 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:10.194+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:10.483 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:10.327+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:10.483 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:10.327+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:10.483 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:10.327+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:10.483 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:10.328+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:10.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:10.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:10.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:10.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:10.482+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:11.512 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:11.188+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:11.515 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:11.188+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:11.515 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:11.188+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:11.515 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:11.189+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:11.515 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:11.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:11.515 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:11.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:11.515 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:11.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:11.515 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:11.307+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:11.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:11.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:11.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:11.512+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:12.521 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:12.236+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:12.522 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:12.236+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:12.522 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:12.236+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:12.522 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:12.237+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:12.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:12.265+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:12.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:12.265+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:12.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:12.265+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:12.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:12.266+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:12.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:12.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:12.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:12.522+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:13.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:13.057+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1417 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:13.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:13.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:13.217+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:13.217+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:13.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:13.264+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:13.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:13.264+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:13.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:13.264+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:13.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:13.265+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:13.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:13.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:13.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:13.524+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:14.529 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:14.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:14.529 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:14.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:14.529 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:14.234+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:14.530 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:14.235+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:14.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:14.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:14.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:14.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:14.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:14.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:14.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:14.300+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:14.530 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:14.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:14.530 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:14.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:14.530 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:14.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:14.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:14.529+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:15.521 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:15.206+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:15.521 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:15.206+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:15.521 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:15.206+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:15.522 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:15.207+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:15.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:15.281+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:15.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:15.281+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:15.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:15.281+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:15.522 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:15.282+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:15.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:15.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:15.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:15.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:15.521+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:16.554 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:16.254+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:16.554 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:16.254+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:16.554 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:16.254+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:16.554 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:16.255+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:16.555 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:16.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:16.555 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:16.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:16.555 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:16.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:16.555 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:16.297+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:16.828 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:16.553+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:16.829 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:16.553+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:16.829 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:16.553+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:16.829 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:16.554+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:17.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:17.295+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:17.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:17.295+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:17.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:17.295+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:17.530 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:17.296+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:17.530 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:17.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:17.530 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:17.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:17.530 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:17.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:17.530 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:17.219+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:17.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:17.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:17.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:17.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:17.530+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:18.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:18.058+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1513 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:18.283+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:18.283+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:18.283+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:18.284+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:18.254+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:18.254+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:18.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:18.254+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:18.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:18.254+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:18.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:18.512+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:18.512+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:18.512+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:18.513+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:19.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:19.304+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:19.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:19.304+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:19.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:19.304+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:19.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:19.305+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:19.497+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:19.497+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:19.497+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:19.498+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:19.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:19.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:19.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:19.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:19.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:19.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:19.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:19.292+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:20.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:20.262+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:20.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:20.262+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:20.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:20.262+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:20.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:20.262+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:20.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:20.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:20.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:20.502+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:20.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:20.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:20.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:20.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:20.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:20.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:20.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:20.318+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:21.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:21.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:21.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:21.306+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:21.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:21.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:21.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:21.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:21.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:21.520+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:21.319+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:21.319+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:21.319+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:21.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:21.320+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:22.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:22.336+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:22.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:22.336+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:22.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:22.336+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:22.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:22.337+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:22.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:22.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:22.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:22.347+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:22.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:22.554+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:22.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:22.554+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:22.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:22.554+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:22.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:22.555+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:23.351 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:23.059+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1608 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:23.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:23.289+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:23.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:23.289+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:23.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:23.289+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:23.352 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:23.290+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:23.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:23.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:23.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:23.522+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:23.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:23.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:23.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:23.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:23.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:23.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:23.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:23.351+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:24.544 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:24.395+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:24.544 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:24.395+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:24.544 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:24.395+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:24.544 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:24.396+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:24.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:24.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:24.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:24.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:24.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:24.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:24.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:24.256+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:24.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:24.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:24.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:24.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:24.544+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:25.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:25.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:25.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:25.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:25.522+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:25.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:25.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:25.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:25.349+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:25.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:25.275+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:25.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:25.275+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:25.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:25.275+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:25.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:25.276+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:26.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:26.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:26.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:26.499+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:26.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:26.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:26.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:26.364+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:26.308+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:26.308+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:26.308+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:26.309+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:27.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:27.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:27.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:27.462+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:27.326+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:27.326+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:27.326+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:27.327+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:27.314+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:27.314+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:27.314+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:27.314+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:28.311 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:28.059+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1701 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:28.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:28.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:28.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:28.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:28.504+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:28.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:28.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:28.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:28.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:28.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:28.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:28.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:28.346+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:28.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:28.311+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:28.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:28.311+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:28.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:28.311+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:28.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:28.312+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:29.539+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:29.539+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:29.539+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:29.539+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:29.360+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:29.360+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:29.360+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:29.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:29.361+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:29.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:29.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:29.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:29.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:29.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:29.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:29.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:29.342+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:30.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:30.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:30.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:30.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:30.504+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:30.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:30.380+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:30.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:30.380+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:30.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:30.380+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:30.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:30.381+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:30.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:30.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:30.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:30.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:30.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:30.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:30.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:30.356+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:31.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:31.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:31.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:31.518+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:31.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:31.328+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:31.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:31.328+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:31.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:31.328+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:31.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:31.330+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:31.395+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:31.395+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:31.395+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:31.396+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:32.538 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:32.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:32.538 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:32.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:32.538 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:32.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:32.538 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:32.377+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:32.539 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:32.363+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:32.539 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:32.363+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:32.539 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:32.363+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:32.539 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:32.364+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:32.537+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:32.537+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:32.537+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:32.538+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:33.344 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:33.060+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1795 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:33.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:33.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:33.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:33.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:33.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:33.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:33.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:33.502+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:33.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:33.394+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:33.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:33.394+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:33.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:33.394+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:33.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:33.395+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:33.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:33.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:33.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:33.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:33.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:33.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:33.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:33.344+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:34.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:34.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:34.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:34.504+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:34.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:34.425+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:34.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:34.425+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:34.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:34.425+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:34.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:34.426+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:34.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:34.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:34.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:34.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:34.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:34.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:34.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:34.367+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:35.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:35.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:35.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:35.462+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:35.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:35.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:35.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:35.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:35.457+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:35.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:35.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:35.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:35.332+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:36.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:36.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:36.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:36.416+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:36.429+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:36.429+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:36.429+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:36.430+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:36.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:36.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:36.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:36.308+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:37.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:37.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:37.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:37.415+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:37.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:37.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:37.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:37.399+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:37.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:37.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:37.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:37.281+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:38.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:38.060+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1891 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:38.279+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:38.279+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:38.279+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:38.280+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:38.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:38.383+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:38.383+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:38.383+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:38.384+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:38.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:38.359+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:38.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:38.359+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:38.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:38.359+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:38.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:38.360+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:39.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:39.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:39.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:39.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:39.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:39.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:39.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:39.385+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:39.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:39.324+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:39.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:39.324+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:39.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:39.324+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:39.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:39.325+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:39.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:39.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:39.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:39.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:39.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:39.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:39.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:39.307+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:40.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:40.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:40.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:40.358+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:40.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:40.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:40.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:40.352+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:40.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:40.261+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:40.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:40.261+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:40.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:40.261+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:40.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:40.262+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:41.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:41.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:41.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:41.398+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:41.342+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:41.342+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:41.342+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:41.343+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:41.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:41.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:41.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:41.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:41.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:41.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:41.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:41.300+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:42.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:42.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:42.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:42.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:42.425+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:42.314+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:42.314+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:42.314+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:42.316+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:42.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:42.283+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:42.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:42.283+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:42.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:42.283+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:42.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:42.284+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:43.061+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 1985 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:43.261+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:43.261+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:43.261+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:43.262+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:43.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:43.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:43.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:43.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:43.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:43.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:43.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:43.284+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:43.455+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:43.455+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:43.455+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:43.455+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:44.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:44.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:44.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:44.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:44.335+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:44.471+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:44.471+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:44.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:44.471+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:44.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:44.472+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:44.256+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:44.256+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:44.256+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:44.257+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:45.281+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:45.281+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:45.281+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:45.282+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:45.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:45.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:45.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:45.505+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:45.325+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:45.325+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:45.325+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:45.326+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:46.252+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:46.252+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:46.252+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:46.253+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:46.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:46.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:46.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:46.469+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:46.282+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:46.282+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:46.282+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:46.283+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:47.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:47.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:47.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:47.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:47.272+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:47.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:47.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:47.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:47.305+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:47.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:47.478+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:47.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:47.478+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:47.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:47.478+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:47.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:47.479+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:48.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:48.061+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 2078 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:48.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:48.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:48.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:48.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:48.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:48.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:48.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:48.272+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:48.353+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:48.353+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:48.353+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:48.354+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:48.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:48.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:48.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:48.439+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:49.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:49.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:49.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:49.485+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:49.395+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:49.395+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:49.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:49.395+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:49.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:49.395+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:49.256+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:49.256+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:49.256+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:49.257+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:50.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:50.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:50.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:50.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:50.297+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:50.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:50.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:50.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:50.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:50.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:50.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:50.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:50.511+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:50.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:50.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:50.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:50.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:50.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:50.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:50.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:50.378+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:51.559 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:51.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:51.559 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:51.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:51.559 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:51.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:51.559 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:51.332+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:51.560 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:51.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:51.560 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:51.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:51.560 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:51.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:51.560 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:51.255+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:51.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:51.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:51.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:51.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:51.559+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:52.560 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:52.321+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:52.561 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:52.321+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:52.561 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:52.321+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:52.561 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:52.322+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:52.561 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:52.302+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:52.561 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:52.302+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:52.561 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:52.302+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:52.561 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:52.303+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:52.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:52.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:52.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:52.560+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:53.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:53.063+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 2172 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:53.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:53.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:53.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:53.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:53.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:53.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:53.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:53.256+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:53.692 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:53.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:53.692 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:53.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:53.692 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:53.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:53.692 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:53.531+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:53.692 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:53.359+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:53.692 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:53.359+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:53.693 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:53.359+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:53.693 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:53.359+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:54.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:54.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:54.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:54.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:54.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:54.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:54.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:54.350+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:54.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:54.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:54.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:54.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:54.483+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:54.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:54.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:54.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:54.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:54.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:54.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:54.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:54.280+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:55.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:55.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:55.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:55.456+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:55.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:55.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:55.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:55.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:55.392+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:55.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:55.260+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:55.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:55.260+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:55.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:55.260+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:55.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:55.260+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:56.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:56.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:56.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:56.447+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:56.366+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:56.366+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:56.366+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:56.368+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:56.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:56.239+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:56.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:56.239+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:56.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:56.239+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:56.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:56.239+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:57.409+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:57.409+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:57.409+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:57.410+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:57.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:57.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:57.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:57.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:57.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:57.405+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:57.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:57.200+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:57.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:57.200+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:57.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:57.200+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:57.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:57.201+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:58.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:22:58.063+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 2265 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:22:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:58.160+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:58.160+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:58.160+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:58.161+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:58.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:58.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:58.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:58.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:58.433+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:58.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:58.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:58.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:58.359+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:59.413 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:59.329+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:59.413 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:59.329+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:59.413 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:59.329+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:22:59.413 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:22:59.329+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:59.414 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:59.123+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:59.414 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:59.123+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:59.414 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:59.123+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:22:59.414 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:22:59.123+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:22:59.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:59.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:59.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:59.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:22:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:22:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:22:59.413+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:00.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:00.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:00.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:00.372+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:00.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:00.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:00.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:00.374+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:00.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:00.148+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:00.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:00.148+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:00.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:00.148+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:00.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:00.148+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:01.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:01.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:01.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:01.103+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:01.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:01.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:01.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:01.402+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:01.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:01.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:01.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:01.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:01.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:01.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:01.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:01.418+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:02.072+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:02.072+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:02.072+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:02.073+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:02.399+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:02.399+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:02.399+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:02.400+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:02.388+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:02.388+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:02.388+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:02.389+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:03.353 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:03.064+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 2360 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:03.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:03.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:03.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:03.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:03.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:03.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:03.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:03.056+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:03.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:03.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:03.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:03.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:03.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:03.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:03.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:03.353+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:03.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:03.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:03.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:03.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:03.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:03.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:03.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:03.384+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:04.346 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:04.085+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:04.347 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:04.085+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:04.347 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:04.085+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:04.347 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:04.086+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:04.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:04.394+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:04.394+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:04.394+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:04.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:04.395+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:04.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:04.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:04.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:04.345+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:05.084+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:05.084+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:05.084+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:05.085+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:05.332+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:05.332+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:05.332+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:05.333+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:05.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:05.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:05.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:05.418+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:06.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:06.037+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:06.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:06.038+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:06.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:06.038+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:06.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:06.039+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:06.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:06.350+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:06.350+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:06.350+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:06.351+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:06.440+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:06.440+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:06.440+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:06.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:06.440+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:07.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:06.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:06.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:06.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:06.991+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:07.363+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:07.363+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:07.363+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:07.364+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:07.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:07.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:07.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:07.412+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:08.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:08.064+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 2453 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:07.980+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:07.980+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:07.980+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:07.981+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:08.367+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:08.367+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:08.367+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:08.368+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:08.367+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:08.367+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:08.367+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:08.368+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:09.325 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:09.015+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:09.325 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:09.015+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:09.325 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:09.015+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:09.325 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:09.015+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:09.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:09.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:09.417+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:09.418+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:09.324+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:09.324+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:09.324+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:09.324+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:10.310 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:10.007+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:10.310 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:10.007+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:10.310 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:10.007+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:10.311 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:10.008+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:10.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:10.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:10.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:10.310+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:10.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:10.446+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:10.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:10.446+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:10.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:10.446+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:10.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:10.447+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:11.326 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:10.965+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:11.326 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:10.965+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:11.326 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:10.965+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:11.326 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:10.966+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:11.325+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:11.325+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:11.325+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:11.325+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:11.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:11.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:11.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:11.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:11.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:11.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:11.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:11.458+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:12.288 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:11.935+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:12.289 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:11.935+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:12.289 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:11.935+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:12.289 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:11.935+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:12.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:12.287+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:12.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:12.287+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:12.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:12.287+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:12.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:12.288+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:12.497+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:12.497+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:12.497+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:12.498+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:13.066 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:12.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:13.066 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:12.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:13.066 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:12.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:13.067 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:12.982+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:13.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:13.065+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 2550 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:13.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:13.297+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:13.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:13.297+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:13.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:13.297+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:13.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:13.298+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:13.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:13.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:13.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:13.453+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:14.348 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:14.022+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:14.348 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:14.022+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:14.349 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:14.022+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:14.349 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:14.022+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:14.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:14.347+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:14.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:14.347+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:14.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:14.347+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:14.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:14.348+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:14.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:14.498+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:14.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:14.498+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:14.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:14.498+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:14.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:14.499+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:15.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:15.072+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:15.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:15.072+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:15.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:15.072+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:15.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:15.073+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:15.329+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:15.329+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:15.329+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:15.330+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:15.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:15.492+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:15.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:15.492+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:15.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:15.492+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:15.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:15.492+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:16.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:16.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:16.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:16.043+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:16.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:16.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:16.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:16.371+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:16.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:16.486+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:16.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:16.486+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:16.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:16.486+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:16.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:16.487+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:17.352 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:17.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:17.352 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:17.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:17.352 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:17.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:17.352 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:17.055+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:17.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:17.350+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:17.350+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:17.351+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:17.352+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:17.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:17.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:17.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:17.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:17.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:17.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:17.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:17.459+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:18.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:18.066+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 2644 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:18.035+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:18.035+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:18.035+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:18.036+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:18.453+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:18.453+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:18.453+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:18.454+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:18.361+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:18.361+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:18.361+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:18.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:18.362+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:19.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:19.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:19.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:19.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:19.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:19.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:19.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:19.063+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:19.493+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:19.494+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:19.494+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:19.494+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:19.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:19.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:19.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:19.313+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:20.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:20.038+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:20.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:20.038+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:20.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:20.038+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:20.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:20.039+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:20.313+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:20.313+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:20.313+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:20.313+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:20.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:20.499+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:20.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:20.499+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:20.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:20.499+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:20.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:20.500+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:21.332 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:21.069+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:21.333 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:21.069+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:21.333 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:21.069+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:21.333 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:21.070+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:21.332+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:21.332+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:21.332+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:21.332+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:21.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:21.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:21.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:21.509+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:22.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:22.043+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:22.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:22.043+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:22.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:22.043+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:22.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:22.044+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:22.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:22.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:22.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:22.312+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:22.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:22.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:22.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:22.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:22.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:22.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:22.469+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:23.068 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:23.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:23.068 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:23.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:23.068 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:23.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:23.068 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:23.029+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:23.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:23.066+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 2737 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:23.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:23.276+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:23.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:23.276+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:23.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:23.276+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:23.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:23.277+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:23.477+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:23.477+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:23.477+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:23.478+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:24.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:23.989+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:24.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:23.989+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:24.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:23.989+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:24.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:23.990+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:24.315+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:24.315+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:24.315+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:24.316+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:24.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:24.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:24.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:24.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:24.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:24.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:24.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:24.453+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:25.290 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:24.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:25.290 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:24.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:25.290 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:24.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:25.290 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:24.971+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:25.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:25.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:25.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:25.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:25.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:25.453+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:25.288+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:25.288+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:25.288+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:25.289+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:26.309 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:25.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:26.309 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:25.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:26.309 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:25.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:26.309 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:25.946+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:26.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:26.462+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:26.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:26.462+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:26.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:26.462+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:26.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:26.463+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:26.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:26.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:26.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:26.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:26.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:26.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:26.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:26.309+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:27.302 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:26.898+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:27.302 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:26.898+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:27.302 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:26.898+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:27.302 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:26.899+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:27.301+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:27.301+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:27.301+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:27.301+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:27.485+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:27.485+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:27.485+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:27.485+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:28.191 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:28.067+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 2828 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:28.191 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:27.938+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:28.191 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:27.938+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:28.192 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:27.938+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:28.192 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:27.939+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:28.447 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:28.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:28.447 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:28.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:28.447 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:28.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:28.447 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:28.317+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:28.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:28.446+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:28.446+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:28.446+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:28.447+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:29.335 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:28.943+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:29.335 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:28.943+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:29.335 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:28.943+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:29.335 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:28.943+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:29.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:29.333+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:29.333+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:29.333+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:29.334+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:29.420+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:29.420+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:29.420+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:29.421+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:30.319 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:29.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:30.319 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:29.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:30.320 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:29.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:30.320 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:29.991+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:30.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:30.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:30.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:30.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:30.399+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:30.317+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:30.317+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:30.317+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:30.318+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:31.321 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:30.950+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:31.322 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:30.950+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:31.322 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:30.950+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:31.322 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:30.951+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:31.415+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:31.415+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:31.415+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:31.416+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:31.319+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:31.319+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:31.319+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:31.320+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:32.343 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:31.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:32.343 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:31.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:32.343 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:31.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:32.343 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:31.923+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:32.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:32.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:32.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:32.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:32.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:32.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:32.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:32.384+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:32.341+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:32.341+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:32.341+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:32.342+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:33.069 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:32.932+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:33.069 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:32.932+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:33.069 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:32.932+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:33.069 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:32.932+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:33.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:33.067+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 2923 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:33.628 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:33.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:33.628 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:33.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:33.628 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:33.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:33.628 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:33.381+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:33.628 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:33.366+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:33.628 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:33.366+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:33.628 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:33.366+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:33.629 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:33.366+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:34.325 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:33.962+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:34.326 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:33.962+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:34.326 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:33.962+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:34.326 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:33.963+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:34.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:34.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:34.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:34.387+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:34.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:34.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:34.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:34.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:34.324+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:35.328 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:34.991+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:35.329 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:34.991+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:35.329 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:34.991+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:35.329 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:34.992+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:35.399+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:35.399+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:35.399+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:35.400+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:35.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:35.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:35.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:35.328+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:36.349 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:35.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:36.349 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:35.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:36.349 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:35.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:36.349 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:36.000+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:36.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:36.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:36.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:36.415+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:36.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:36.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:36.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:36.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:36.349+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:37.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:37.039+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:37.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:37.039+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:37.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:37.039+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:37.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:37.040+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:37.329+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:37.329+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:37.329+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:37.330+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:37.393+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:37.393+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:37.393+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:37.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:37.394+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:38.347 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:38.068+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 3015 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:38.347 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:37.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:38.347 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:37.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:38.347 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:37.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:38.347 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:37.996+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:38.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:38.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:38.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:38.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:38.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:38.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:38.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:38.347+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:38.380+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:38.380+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:38.380+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:38.381+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:38.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:38.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:38.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:38.982+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:39.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:39.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:39.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:39.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:39.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:39.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:39.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:39.366+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:39.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:39.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:39.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:39.370+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:40.350 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:39.953+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:40.350 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:39.953+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:40.350 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:39.953+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:40.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:39.953+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:40.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:40.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:40.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:40.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:40.350+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:40.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:40.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:40.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:40.366+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:41.304 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:40.958+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:41.305 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:40.958+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:41.305 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:40.958+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:41.305 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:40.958+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:41.303+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:41.303+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:41.303+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:41.304+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:41.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:41.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:41.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:41.374+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:42.276 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:41.967+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:42.276 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:41.967+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:42.276 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:41.967+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:42.276 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:41.968+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:42.274+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:42.274+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:42.274+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:42.275+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:42.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:42.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:42.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:42.379+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:43.071 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:42.953+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:43.071 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:42.953+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:43.071 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:42.953+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:43.071 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:42.954+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:43.069+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 3109 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:43.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:43.252+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:43.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:43.252+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:43.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:43.252+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:43.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:43.252+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:43.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:43.407+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:43.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:43.407+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:43.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:43.407+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:43.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:43.408+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:44.258 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:43.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:44.258 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:43.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:44.258 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:43.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:44.258 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:43.942+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:44.446+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:44.447+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:44.447+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:44.447+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:44.256+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:44.256+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:44.256+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:44.257+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:45.251 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:44.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:45.251 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:44.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:45.251 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:44.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:45.251 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:44.980+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:45.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:45.250+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:45.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:45.250+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:45.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:45.250+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:45.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:45.251+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:45.462+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:45.462+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:45.462+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:45.463+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:46.280 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:46.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:46.280 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:46.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:46.280 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:46.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:46.280 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:46.019+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:46.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:46.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:46.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:46.279+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:46.434+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:46.434+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:46.434+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:46.435+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:47.321 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:47.019+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:47.322 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:47.019+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:47.322 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:47.019+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:47.322 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:47.019+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:47.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:47.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:47.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:47.413+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:47.320+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:47.320+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:47.320+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:47.321+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:48.319 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:48.070+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 3199 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:48.319 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:47.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:48.319 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:47.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:48.319 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:47.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:48.319 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:47.974+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:48.318+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:48.318+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:48.318+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:48.318+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:48.426+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:48.426+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:48.426+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:48.427+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:49.282 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:48.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:49.282 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:48.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:49.282 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:48.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:49.282 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:48.997+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:49.281+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:49.281+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:49.281+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:49.282+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:49.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:49.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:49.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:49.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:49.411+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:50.328 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:49.997+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:50.328 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:49.997+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:50.328 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:49.997+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:50.328 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:49.997+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:50.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:50.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:50.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:50.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:50.327+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:50.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:50.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:50.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:50.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:50.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:50.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:50.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:50.387+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:51.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:51.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:51.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:51.029+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:51.649 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:51.375+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:51.650 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:51.375+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:51.650 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:51.375+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:51.650 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:51.376+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:51.650 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:51.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:51.650 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:51.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:51.650 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:51.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:51.650 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:51.371+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:52.341 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:52.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:52.341 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:52.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:52.341 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:52.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:52.341 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:52.057+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:52.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:52.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:52.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:52.341+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:52.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:52.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:52.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:52.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:52.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:52.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:52.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:52.350+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:53.337 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:53.071+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 3294 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:53.337 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:53.089+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:53.337 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:53.089+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:53.338 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:53.089+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:53.338 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:53.089+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:53.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:53.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:53.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:53.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:53.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:53.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:53.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:53.351+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:53.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:53.336+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:53.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:53.336+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:53.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:53.336+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:53.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:53.337+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:54.327 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:54.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:54.327 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:54.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:54.327 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:54.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:54.327 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:54.310+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:54.328 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:54.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:54.328 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:54.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:54.328 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:54.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:54.328 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:54.063+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:54.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:54.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:54.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:54.327+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:55.096+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:55.096+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:55.096+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:55.097+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:55.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:55.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:55.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:55.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:55.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:55.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:55.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:55.316+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:55.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:55.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:55.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:55.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:55.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:55.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:55.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:55.266+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:56.123+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:56.123+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:56.123+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:56.124+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:56.354+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:56.354+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:56.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:56.354+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:56.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:56.355+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:56.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:56.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:56.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:56.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:56.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:56.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:56.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:56.305+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:57.383 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:57.261+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:57.383 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:57.261+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:57.383 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:57.261+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:57.383 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:57.262+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:57.383 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:57.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:57.383 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:57.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:57.383 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:57.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:57.383 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:57.127+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:57.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:57.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:57.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:57.382+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:58.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:23:58.071+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 3386 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:23:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:58.285+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:58.285+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:58.285+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:58.286+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:58.162+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:58.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:58.162+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:58.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:58.162+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:58.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:58.162+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:58.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:58.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:58.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:58.363+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:59.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:59.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:59.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:23:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:23:59.349+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:59.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:59.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:59.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:23:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:23:59.265+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:23:59.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:59.212+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:59.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:59.212+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:59.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:59.212+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:23:59.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:23:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:23:59.213+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:00.396+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:00.396+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:00.396+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:00.397+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:00.216+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:00.216+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:00.216+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:00.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:00.217+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:00.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:00.226+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:00.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:00.226+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:00.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:00.226+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:00.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:00.227+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:01.198+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:01.198+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:01.198+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:01.199+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:01.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:01.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:01.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:01.426+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:01.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:01.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:01.255+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:01.255+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:02.459 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:02.194+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:02.459 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:02.194+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:02.459 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:02.194+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:02.459 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:02.195+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:02.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:02.244+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:02.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:02.244+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:02.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:02.244+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:02.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:02.245+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:02.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:02.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:02.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:02.459+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:03.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:03.072+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 3480 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:03.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:03.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:03.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:03.207+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:03.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:03.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:03.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:03.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:03.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:03.280+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:03.422+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:03.422+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:03.422+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:03.422+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:04.416+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:04.416+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:04.416+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:04.416+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:04.261+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:04.261+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:04.261+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:04.261+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:04.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:04.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:04.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:04.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:04.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:04.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:04.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:04.243+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:05.466 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:05.205+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:05.467 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:05.205+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:05.467 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:05.205+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:05.467 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:05.205+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:05.467 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:05.276+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:05.467 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:05.276+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:05.467 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:05.276+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:05.467 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:05.276+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:05.465+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:05.465+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:05.465+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:05.466+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:06.499 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:06.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:06.499 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:06.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:06.499 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:06.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:06.499 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:06.219+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:06.499 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:06.269+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:06.499 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:06.269+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:06.500 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:06.269+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:06.500 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:06.270+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:06.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:06.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:06.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:06.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:06.499+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:07.524 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:07.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:07.524 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:07.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:07.524 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:07.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:07.524 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:07.248+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:07.525 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:07.298+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:07.525 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:07.298+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:07.525 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:07.298+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:07.525 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:07.299+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:07.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:07.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:07.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:07.524+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:08.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:08.073+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 3571 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:08.290+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:08.290+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:08.290+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:08.291+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:08.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:08.319+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:08.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:08.319+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:08.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:08.319+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:08.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:08.320+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:08.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:08.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:08.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:08.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:08.478+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:09.303+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:09.303+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:09.303+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:09.304+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:09.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:09.484+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:09.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:09.484+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:09.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:09.484+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:09.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:09.484+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:09.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:09.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:09.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:09.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:09.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:09.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:09.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:09.359+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:10.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:10.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:10.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:10.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:10.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:10.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:10.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:10.399+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:10.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:10.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:10.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:10.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:10.456+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:10.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:10.325+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:10.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:10.325+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:10.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:10.325+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:10.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:10.326+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:11.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:11.393+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:11.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:11.393+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:11.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:11.393+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:11.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:11.393+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:11.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:11.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:11.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:11.425+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:11.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:11.338+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:11.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:11.338+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:11.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:11.338+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:11.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:11.338+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:12.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:12.388+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:12.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:12.388+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:12.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:12.388+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:12.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:12.389+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:12.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:12.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:12.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:12.387+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:12.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:12.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:12.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:12.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:12.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:12.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:12.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:12.362+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:13.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:13.074+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 3667 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:13.366+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:13.366+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:13.366+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:13.366+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:13.390+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:13.390+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:13.390+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:13.391+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:13.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:13.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:13.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:13.380+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:14.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:14.411+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:14.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:14.411+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:14.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:14.411+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:14.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:14.412+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:14.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:14.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:14.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:14.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:14.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:14.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:14.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:14.380+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:14.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:14.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:14.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:14.331+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:15.333+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:15.333+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:15.333+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:15.334+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:15.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:15.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:15.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:15.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:15.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:15.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:15.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:15.416+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:15.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:15.344+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:15.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:15.344+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:15.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:15.344+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:15.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:15.345+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:16.637 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:16.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:16.637 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:16.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:16.637 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:16.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:16.637 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:16.354+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:16.637 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:16.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:16.638 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:16.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:16.638 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:16.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:16.638 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:16.416+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:16.638 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:16.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:16.638 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:16.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:16.638 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:16.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:16.638 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:16.365+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:17.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:17.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:17.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:17.348+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:17.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:17.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:17.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:17.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:17.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:17.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:17.375+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:17.362+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:17.362+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:17.362+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:17.362+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:18.349 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:18.074+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 3760 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:18.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:18.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:18.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:18.412+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:18.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:18.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:18.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:18.348+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:18.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:18.353+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:18.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:18.353+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:18.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:18.353+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:18.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:18.353+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:19.366+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:19.366+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:19.366+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:19.367+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:19.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:19.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:19.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:19.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:19.412+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:19.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:19.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:19.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:19.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:19.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:19.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:19.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:19.376+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:20.347+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:20.347+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:20.347+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:20.348+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:20.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:20.390+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:20.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:20.390+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:20.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:20.390+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:20.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:20.391+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:20.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:20.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:20.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:20.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:20.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:20.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:20.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:20.422+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:21.652 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:21.385+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:21.652 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:21.386+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:21.653 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:21.386+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:21.653 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:21.387+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:21.653 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:21.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:21.653 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:21.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:21.653 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:21.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:21.653 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:21.358+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:21.653 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:21.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:21.654 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:21.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:21.654 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:21.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:21.654 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:21.435+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:22.780 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:22.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:22.780 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:22.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:22.780 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:22.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:22.780 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:22.364+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:22.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:22.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:22.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:22.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:22.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:22.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:22.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:22.372+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:22.781 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:22.466+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:22.781 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:22.466+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:22.781 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:22.466+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:22.781 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:22.467+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:23.338 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:23.075+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 3854 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:23.360+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:23.360+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:23.360+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:23.361+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:23.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:23.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:23.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:23.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:23.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:23.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:23.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:23.338+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:23.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:23.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:23.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:23.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:23.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:23.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:23.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:23.475+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:24.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:24.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:24.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:24.373+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:24.336+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:24.336+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:24.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:24.336+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:24.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:24.337+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:24.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:24.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:24.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:24.431+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:25.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:25.420+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:25.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:25.420+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:25.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:25.420+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:25.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:25.421+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:25.853 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:25.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:25.853 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:25.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:25.853 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:25.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:25.853 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:25.370+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:25.853 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:25.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:25.853 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:25.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:25.853 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:25.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:25.853 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:25.475+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:26.436+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:26.436+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:26.436+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:26.437+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:26.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:26.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:26.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:26.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:26.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:26.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:26.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:26.370+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:26.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:26.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:26.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:26.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:26.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:26.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:26.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:26.500+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:27.343+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:27.343+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:27.343+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:27.343+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:27.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:27.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:27.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:27.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:27.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:27.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:27.474+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:27.481+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:27.481+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:27.481+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:27.481+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:28.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:28.076+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 3945 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:28.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:28.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:28.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:28.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:28.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:28.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:28.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:28.457+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:28.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:28.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:28.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:28.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:28.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:28.511+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:28.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:28.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:28.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:28.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:28.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:28.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:28.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:28.375+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:29.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:29.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:29.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:29.544+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:29.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:29.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:29.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:29.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:29.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:29.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:29.335+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:29.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:29.433+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:29.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:29.433+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:29.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:29.433+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:29.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:29.434+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:30.556+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:30.556+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:30.556+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:30.557+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:30.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:30.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:30.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:30.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:30.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:30.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:30.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:30.389+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:30.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:30.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:30.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:30.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:30.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:30.337+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:30.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:30.339+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:31.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:31.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:31.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:31.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:31.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:31.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:31.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:31.306+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:31.575 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:31.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:31.575 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:31.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:31.576 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:31.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:31.576 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:31.417+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:31.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:31.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:31.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:31.575+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:32.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:32.259+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:32.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:32.259+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:32.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:32.259+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:32.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:32.260+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:32.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:32.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:32.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:32.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:32.380+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:33.078 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:32.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:33.078 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:32.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:33.078 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:32.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:33.078 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:32.606+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:33.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:33.077+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 4040 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:33.245+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:33.245+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:33.245+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:33.245+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:33.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:33.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:33.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:33.638+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:33.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:33.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:33.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:33.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:33.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:33.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:33.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:33.423+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:34.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:34.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:34.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:34.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:34.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:34.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:34.603 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:34.381+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:34.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:34.264+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:34.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:34.264+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:34.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:34.264+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:34.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:34.264+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:34.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:34.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:34.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:34.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:34.602+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:35.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:35.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:35.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:35.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:35.225+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:35.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:35.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:35.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:35.585+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:35.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:35.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:35.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:35.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:35.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:35.383+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:36.538 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:36.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:36.538 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:36.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:36.538 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:36.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:36.538 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:36.225+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:36.538 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:36.414+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:36.538 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:36.414+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:36.538 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:36.414+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:36.538 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:36.415+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:36.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:36.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:36.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:36.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:36.537+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:37.536 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:37.213+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:37.536 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:37.213+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:37.536 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:37.213+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:37.536 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:37.214+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:37.537 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:37.369+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:37.537 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:37.369+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:37.537 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:37.369+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:37.537 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:37.370+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:37.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:37.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:37.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:37.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:37.536+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:38.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:38.078+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 4132 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:38.232+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:38.232+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:38.232+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:38.233+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:38.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:38.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:38.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:38.508+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:38.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:38.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:38.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:38.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:38.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:38.417+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:39.547 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:39.256+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:39.547 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:39.256+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:39.547 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:39.256+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:39.547 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:39.257+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:39.547 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:39.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:39.547 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:39.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:39.548 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:39.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:39.548 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:39.459+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:39.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:39.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:39.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:39.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:39.547+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:40.571 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:40.292+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:40.571 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:40.292+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:40.571 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:40.292+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:40.571 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:40.293+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:40.571 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:40.495+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:40.571 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:40.495+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:40.572 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:40.495+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:40.572 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:40.495+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:40.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:40.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:40.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:40.570+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:41.592 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:41.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:41.593 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:41.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:41.593 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:41.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:41.593 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:41.324+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:41.593 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:41.447+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:41.593 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:41.447+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:41.593 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:41.447+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:41.593 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:41.447+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:41.591+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:41.591+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:41.591+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:41.592+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:42.565+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:42.565+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:42.565+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:42.566+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:42.446+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:42.446+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:42.446+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:42.447+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:42.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:42.322+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:42.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:42.322+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:42.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:42.322+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:42.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:42.323+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:43.348 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:43.078+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 4226 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:43.599 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:43.457+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:43.599 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:43.457+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:43.599 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:43.457+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:43.599 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:43.458+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:43.599 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:43.597+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:43.599 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:43.597+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:43.599 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:43.597+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:43.600 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:43.347+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:43.600 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:43.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:43.600 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:43.348+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:43.600 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:43.348+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:43.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:43.598+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:44.566 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:44.380+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:44.567 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:44.380+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:44.567 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:44.380+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:44.567 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:44.380+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:44.567 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:44.487+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:44.567 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:44.487+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:44.567 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:44.487+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:44.567 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:44.488+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:44.565+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:44.565+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:44.565+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:44.566+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:45.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:45.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:45.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:45.542+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:45.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:45.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:45.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:45.398+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:45.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:45.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:45.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:45.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:45.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:45.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:45.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:45.506+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:46.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:46.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:46.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:46.544+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:46.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:46.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:46.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:46.351+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:46.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:46.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:46.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:46.505+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:47.537+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:47.537+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:47.537+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:47.538+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:47.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:47.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:47.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:47.340+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:47.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:47.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:47.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:47.554+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:48.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:48.079+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 4318 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:48.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:48.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:48.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:48.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:48.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:48.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:48.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:48.314+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:48.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:48.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:48.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:48.584+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:48.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:48.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:48.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:48.529+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:49.554+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:49.554+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:49.554+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:49.554+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:49.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:49.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:49.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:49.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:49.331+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:49.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:49.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:49.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:49.561+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:50.514+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:50.514+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:50.514+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:50.515+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:50.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:50.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:50.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:50.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:50.359+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:50.556+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:50.556+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:50.556+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:50.557+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:51.654 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:51.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:51.654 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:51.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:51.654 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:51.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:51.654 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:51.378+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:51.655 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:51.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:51.655 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:51.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:51.655 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:51.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:51.655 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:51.510+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:51.655 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:51.594+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:51.656 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:51.594+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:51.656 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:51.594+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:51.656 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:51.594+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:52.632 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:52.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:52.632 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:52.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:52.632 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:52.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:52.632 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:52.464+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:52.632 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:52.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:52.632 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:52.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:52.632 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:52.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:52.633 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:52.366+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:53.082 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:52.631+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:53.082 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:52.631+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:53.082 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:52.631+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:53.082 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:52.631+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:53.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:53.080+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 4413 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:53.326+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:53.326+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:53.326+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:53.327+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:53.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:53.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:53.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:53.452+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:53.651+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:53.651+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:53.651+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:53.652+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:54.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:54.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:54.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:54.452+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:54.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:54.299+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:54.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:54.299+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:54.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:54.299+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:54.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:54.299+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:54.698+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:54.698+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:54.698+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:54.699+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:55.429+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:55.429+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:55.429+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:55.430+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:55.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:55.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:55.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:55.284+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:55.745+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:55.745+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:55.745+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:55.746+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:56.572 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:56.469+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:56.572 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:56.469+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:56.572 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:56.469+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:56.572 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:56.470+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:56.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:56.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:56.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:56.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:56.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:56.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:56.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:56.247+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:56.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:56.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:56.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:56.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:56.780+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:57.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:57.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:57.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:57.451+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:57.215+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:57.215+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:57.215+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:57.216+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:58.083 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:57.823+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:58.083 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:57.823+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:58.083 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:57.823+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:58.084 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:57.824+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:58.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:24:58.081+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 4513 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:24:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:58.223+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:58.223+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:58.223+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:58.224+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:58.408+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:58.408+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:58.408+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:58.409+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:59.262 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:58.869+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:59.262 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:58.869+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:59.262 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:58.869+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:24:59.262 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:58.870+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:59.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:59.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:59.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:59.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:59.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:59.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:24:59.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:24:59.416+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:24:59.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:59.259+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:59.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:59.259+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:59.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:59.259+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:24:59.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:24:59.260+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:00.225 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:59.899+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:00.225 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:59.899+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:00.226 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:59.899+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:00.226 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:24:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:24:59.900+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:00.223+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:00.223+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:00.223+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:00.224+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:00.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:00.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:00.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:00.434+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:01.207 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:00.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:01.207 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:00.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:01.207 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:00.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:01.207 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:00.946+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:01.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:01.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:01.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:01.418+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:01.205+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:01.205+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:01.205+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:01.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:01.206+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:02.216+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:02.216+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:02.216+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:02.217+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:01.977+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:01.977+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:01.977+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:01.977+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:02.716 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:02.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:02.716 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:02.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:02.716 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:02.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:02.716 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:02.374+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:03.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:03.081+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 4611 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:03.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:03.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:03.185+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:03.186+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:02.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:02.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:03.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:02.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:03.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:02.980+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:03.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:03.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:03.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:03.363+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:04.319 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:04.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:04.320 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:04.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:04.320 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:04.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:04.320 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:04.208+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:04.320 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:04.017+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:04.320 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:04.017+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:04.320 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:04.017+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:04.320 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:04.018+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:04.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:04.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:04.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:04.317+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:05.306 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:05.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:05.306 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:05.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:05.306 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:05.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:05.306 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:05.185+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:05.306 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:05.005+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:05.306 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:05.005+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:05.306 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:05.005+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:05.307 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:05.006+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:05.305+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:05.305+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:05.305+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:05.306+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:06.008+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:06.008+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:06.008+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:06.009+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:06.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:06.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:06.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:06.328+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:06.148+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:06.148+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:06.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:06.148+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:06.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:06.149+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:06.984+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:06.984+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:06.984+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:06.985+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:07.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:07.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:07.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:07.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:07.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:07.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:07.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:07.156+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:07.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:07.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:07.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:07.373+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:08.341 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:08.082+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 4702 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:08.342 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:08.136+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:08.342 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:08.136+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:08.342 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:08.136+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:08.342 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:08.136+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:08.342 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:08.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:08.342 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:08.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:08.342 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:08.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:08.343 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:08.024+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:08.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:08.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:08.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:08.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:08.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:08.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:08.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:08.341+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:09.141 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:08.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:09.141 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:08.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:09.141 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:08.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:09.142 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:08.980+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:09.320+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:09.320+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:09.320+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:09.321+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:09.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:09.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:09.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:09.141+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:10.298 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:10.174+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:10.299 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:10.174+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:10.299 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:10.174+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:10.299 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:10.175+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:10.299 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:09.944+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:10.299 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:09.944+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:10.299 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:09.944+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:10.299 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:09.945+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:10.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:10.297+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:10.297+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:10.297+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:10.298+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:11.295 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:11.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:11.296 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:11.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:11.296 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:11.207+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:11.296 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:11.208+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:11.296 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:10.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:11.296 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:10.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:11.296 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:10.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:11.296 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:10.990+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:11.294+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:11.294+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:11.294+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:11.295+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:12.199 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:11.988+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:12.199 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:11.988+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:12.199 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:11.988+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:12.199 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:11.989+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:12.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:12.301+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:12.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:12.301+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:12.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:12.301+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:12.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:12.302+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:12.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:12.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:12.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:12.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:12.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:12.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:12.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:12.199+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:13.084 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:13.002+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:13.085 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:13.002+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:13.085 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:13.002+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:13.085 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:13.003+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:13.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:13.083+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 4798 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:13.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:13.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:13.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:13.205+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:13.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:13.305+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:13.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:13.305+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:13.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:13.305+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:13.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:13.306+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:14.308 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:14.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:14.308 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:14.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:14.308 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:14.246+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:14.308 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:14.247+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:14.308 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:14.014+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:14.309 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:14.014+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:14.309 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:14.014+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:14.310 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:14.015+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:14.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:14.307+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:14.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:14.307+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:14.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:14.307+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:14.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:14.308+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:15.254 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:15.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:15.254 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:15.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:15.254 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:15.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:15.255 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:15.063+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:15.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:15.253+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:15.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:15.253+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:15.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:15.253+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:15.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:15.254+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:15.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:15.283+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:15.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:15.283+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:15.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:15.283+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:15.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:15.284+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:16.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:16.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:16.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:16.043+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:16.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:16.246+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:16.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:16.246+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:16.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:16.246+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:16.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:16.247+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:16.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:16.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:16.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:16.215+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:17.283 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:17.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:17.283 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:17.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:17.283 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:17.214+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:17.283 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:17.215+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:17.284 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:16.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:17.284 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:16.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:17.284 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:16.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:17.284 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:16.997+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:17.282+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:17.282+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:17.282+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:17.283+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:18.332 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:18.084+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 4903 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:18.332 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:18.171+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:18.332 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:18.171+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:18.332 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:18.171+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:18.332 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:18.172+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:18.333 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:17.964+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:18.333 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:17.964+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:18.333 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:17.964+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:18.333 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:17.965+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:18.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:18.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:18.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:18.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:18.331+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:19.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:19.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:19.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:19.197+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:18.993+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:18.993+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:18.993+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:18.994+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:19.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:19.363+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:19.363+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:19.363+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:20.169+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:20.169+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:20.169+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:20.169+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:20.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:20.041+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:20.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:20.041+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:20.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:20.041+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:20.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:20.042+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:20.412+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:20.412+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:20.412+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:20.413+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:21.177+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:21.177+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:21.177+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:21.178+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:21.009+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:21.009+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:21.009+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:21.010+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:21.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:21.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:21.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:21.462+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:21.969+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:21.969+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:21.969+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:21.970+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:22.183+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:22.183+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:22.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:22.183+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:22.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:22.184+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:22.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:22.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:22.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:22.435+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:23.086 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:22.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:23.086 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:22.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:23.086 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:22.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:23.086 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:22.925+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:23.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:23.085+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 4997 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:23.141+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:23.141+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:23.141+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:23.142+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:23.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:23.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:23.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:23.459+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:24.220 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:24.108+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:24.220 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:24.108+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:24.220 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:24.108+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:24.220 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:24.108+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:24.220 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:23.877+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:24.221 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:23.877+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:24.221 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:23.877+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:24.221 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:23.877+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:24.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:24.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:24.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:24.432+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:25.149 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:24.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:25.149 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:24.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:25.149 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:24.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:25.149 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:24.866+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:25.450 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:25.147+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:25.451 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:25.147+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:25.451 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:25.147+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:25.451 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:25.149+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:25.449+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:25.449+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:25.449+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:25.450+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:25.869+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:25.869+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:25.869+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:25.869+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:26.111+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:26.111+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:26.111+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:26.112+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:26.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:26.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:26.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:26.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:26.475+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:26.917+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:26.917+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:26.917+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:26.917+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:27.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:27.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:27.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:27.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:27.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:27.075+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:27.494+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:27.494+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:27.494+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:27.495+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:28.210 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:28.086+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 5088 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:28.210 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:27.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:28.210 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:27.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:28.210 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:27.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:28.210 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:27.929+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:28.211 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:28.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:28.211 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:28.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:28.211 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:28.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:28.211 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:28.104+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:28.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:28.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:28.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:28.503+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:29.134+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:29.134+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:29.134+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:29.134+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:28.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:28.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:29.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:28.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:29.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:28.945+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:29.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:29.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:29.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:29.530+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:30.134+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:30.134+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:30.134+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:30.135+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:30.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:29.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:30.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:29.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:30.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:29.990+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:30.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:29.991+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:30.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:30.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:30.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:30.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:30.567+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:31.139+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:31.139+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:31.139+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:31.140+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:30.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:30.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:30.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:30.996+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:31.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:31.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:31.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:31.567+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:32.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:32.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:32.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:32.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:32.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:32.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:32.104+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:32.034+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:32.034+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:32.034+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:32.034+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:32.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:32.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:32.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:32.523+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:33.088 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:32.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:33.088 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:32.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:33.088 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:32.996+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:33.088 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:32.997+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:33.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:33.087+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 5188 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:33.101+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:33.101+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:33.101+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:33.102+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:33.568+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:33.568+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:33.568+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:33.568+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:34.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:34.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:34.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:34.028+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:34.129+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:34.129+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:34.129+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:34.130+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:34.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:34.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:34.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:34.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:34.589+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:35.006+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:35.006+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:35.006+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:35.007+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:35.108+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:35.108+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:35.108+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:35.109+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:35.968 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:35.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:35.968 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:35.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:35.969 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:35.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:35.969 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:35.638+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:35.967+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:35.967+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:35.967+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:35.968+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:36.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:36.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:36.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:36.141+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:36.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:36.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:36.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:36.602+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:37.012+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:37.012+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:37.012+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:37.013+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:37.139+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:37.139+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:37.139+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:37.140+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:37.557+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:37.557+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:37.557+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:37.558+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:38.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:38.088+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 5294 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:37.995+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:37.995+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:37.995+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:37.996+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:38.120+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:38.120+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:38.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:38.120+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:38.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:38.121+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:38.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:38.567+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:38.567+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:38.567+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:38.568+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:39.019+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:39.019+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:39.019+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:39.019+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:39.076+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:39.076+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:39.076+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:39.077+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:39.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:39.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:39.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:39.604+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:40.000+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:40.000+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:40.000+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:40.000+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:40.050+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:40.050+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:40.050+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:40.051+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:40.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:40.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:40.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:40.575+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:41.040+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:41.040+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:41.040+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:41.040+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:41.053+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:41.053+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:41.053+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:41.054+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:41.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:41.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:41.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:41.547+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:42.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:42.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:42.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:42.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:42.038+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:42.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:42.076+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:42.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:42.076+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:42.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:42.076+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:42.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:42.077+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:42.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:42.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:42.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:42.520+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:43.090 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:43.010+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:43.090 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:43.010+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:43.090 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:43.010+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:43.090 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:43.010+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:43.088+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 5402 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:43.109+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:43.109+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:43.109+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:43.110+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:43.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:43.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:43.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:43.481+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:44.048+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:44.048+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:44.048+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:44.049+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:44.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:44.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:44.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:44.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:44.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:44.094+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:44.512+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:44.512+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:44.512+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:44.512+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:45.038+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:45.038+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:45.038+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:45.039+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:45.080+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:45.080+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:45.080+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:45.081+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:45.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:45.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:45.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:45.482+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:46.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:46.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:46.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:46.102+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:46.019+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:46.019+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:46.019+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:46.020+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:46.443+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:46.443+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:46.443+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:46.444+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:47.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:47.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:47.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:47.018+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:47.139+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:47.139+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:47.139+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:47.140+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:47.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:47.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:47.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:47.457+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:48.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:48.089+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 5493 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:48.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:48.159+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:48.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:48.159+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:48.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:48.159+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:48.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:48.160+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:48.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:48.028+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:48.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:48.028+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:48.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:48.028+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:48.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:48.029+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:48.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:48.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:48.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:48.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:48.506+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:49.133+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:49.133+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:49.133+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:49.134+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:49.056+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:49.056+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:49.056+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:49.057+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:49.469+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:49.469+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:49.469+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:49.470+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:50.090+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:50.090+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:50.090+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:50.091+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:50.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:50.029+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:50.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:50.029+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:50.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:50.029+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:50.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:50.029+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:50.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:50.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:50.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:50.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:50.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:50.443+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:51.061+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:51.061+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:51.061+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:51.061+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:51.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:51.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:51.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:51.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:51.018+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:51.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:51.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:51.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:51.461+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:52.073+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:52.073+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:52.073+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:52.074+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:52.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:51.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:52.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:51.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:52.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:51.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:52.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:51.977+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:52.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:52.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:52.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:52.510+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:53.092 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:52.951+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:53.092 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:52.951+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:53.092 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:52.951+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:53.092 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:52.951+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:53.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:53.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:53.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:53.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:53.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:53.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:53.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:53.027+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:53.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:53.090+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 5587 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:53.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:53.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:53.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:53.543+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:53.943+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:53.943+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:53.943+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:53.944+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:54.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:54.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:54.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:54.030+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:54.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:54.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:54.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:54.582+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:54.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:54.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:54.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:54.978+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:54.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:54.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:54.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:54.983+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:55.565+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:55.565+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:55.565+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:55.565+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:56.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:56.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:56.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:56.014+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:55.974+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:55.974+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:55.974+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:55.975+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:56.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:56.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:56.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:56.526+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:57.045+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:57.045+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:57.045+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:57.045+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:56.948+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:56.948+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:56.948+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:56.949+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:57.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:57.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:57.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:57.521+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:58.214 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:25:58.091+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 5680 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:25:58.214 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:58.046+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:58.214 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:58.046+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:58.214 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:58.046+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:58.214 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:58.046+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:58.215 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:57.956+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:58.215 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:57.956+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:58.215 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:57.956+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:58.215 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:57.957+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:58.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:58.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:58.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:58.492+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:59.024+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:59.024+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:59.024+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:25:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:25:59.025+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:58.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:58.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:58.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:25:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:58.942+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:25:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:59.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:59.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:59.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:59.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:25:59.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:25:59.448+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:00.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:00.002+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:00.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:00.002+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:00.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:00.002+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:00.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:00.004+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:00.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:59.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:00.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:59.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:00.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:59.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:00.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:25:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:25:59.958+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:00.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:00.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:00.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:00.488+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:01.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:01.031+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:01.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:01.031+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:01.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:01.031+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:01.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:01.032+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:00.911+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:00.911+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:00.911+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:01.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:00.912+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:01.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:01.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:01.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:01.506+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:01.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:01.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:01.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:01.996+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:01.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:01.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:01.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:02.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:01.922+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:02.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:02.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:02.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:02.490+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:03.093 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:02.974+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:03.093 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:02.974+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:03.093 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:02.974+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:03.093 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:02.974+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:03.094 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:02.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:03.094 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:02.881+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:03.094 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:02.881+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:03.094 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:02.882+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:03.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:03.092+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 5773 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:03.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:03.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:03.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:03.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:03.490+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:03.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:03.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:03.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:03.976+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:03.927+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:03.927+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:03.927+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:03.928+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:04.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:04.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:04.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:04.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:04.464+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:05.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:05.003+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:05.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:05.003+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:05.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:05.003+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:05.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:05.003+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:04.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:04.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:04.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:04.898+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:05.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:05.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:05.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:05.475+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:06.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:05.968+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:06.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:05.968+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:06.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:05.968+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:06.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:05.969+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:05.868+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:05.868+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:05.868+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:06.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:05.869+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:06.826 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:06.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:06.826 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:06.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:06.826 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:06.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:06.826 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:06.504+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:06.961+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:06.961+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:06.961+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:06.962+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:07.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:06.825+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:07.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:06.825+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:07.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:06.825+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:07.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:06.826+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:07.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:07.553+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:07.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:07.553+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:07.824 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:07.553+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:07.824 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:07.554+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:08.096 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:07.991+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:08.096 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:07.991+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:08.096 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:07.991+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:08.096 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:07.992+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:08.097 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:07.822+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:08.097 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:07.822+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:08.097 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:07.822+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:08.097 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:07.823+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:08.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:08.095+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 5866 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:08.571+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:08.571+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:08.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:08.571+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:08.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:08.572+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:08.859+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:08.859+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:08.859+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:08.860+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:08.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:08.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:08.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:08.997+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:09.812 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:09.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:09.812 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:09.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:09.812 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:09.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:09.812 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:09.542+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:09.811+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:09.811+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:09.811+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:09.811+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:10.016+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:10.016+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:10.016+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:10.017+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:10.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:10.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:10.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:10.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:10.503+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:11.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:10.814+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:11.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:10.814+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:11.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:10.814+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:11.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:10.815+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:11.049+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:11.050+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:11.050+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:11.050+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:11.831 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:11.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:11.832 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:11.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:11.832 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:11.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:11.832 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:11.510+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:12.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:12.065+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:12.065+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:12.065+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:12.065+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:11.830+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:11.830+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:11.830+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:11.831+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:12.799 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:12.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:12.799 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:12.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:12.799 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:12.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:12.799 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:12.488+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:13.097 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:12.798+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:13.097 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:12.798+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:13.097 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:12.798+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:13.097 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:12.799+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:13.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:13.096+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 5960 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:13.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:13.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:13.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:13.098+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:13.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:13.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:13.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:13.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:13.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:13.501+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:13.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:13.502+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:14.067 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:13.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:14.067 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:13.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:14.067 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:13.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:14.067 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:13.787+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:14.066+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:14.066+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:14.066+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:14.066+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:14.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:14.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:14.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:14.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:14.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:14.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:14.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:14.482+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:15.060 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:14.787+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:15.060 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:14.787+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:15.060 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:14.787+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:15.060 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:14.788+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:15.058+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:15.058+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:15.058+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:15.059+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:15.839 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:15.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:15.839 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:15.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:15.839 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:15.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:15.839 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:15.445+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:16.047+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:16.047+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:16.047+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:16.048+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:16.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:15.837+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:16.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:15.837+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:16.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:15.837+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:16.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:15.838+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:16.806 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:16.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:16.806 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:16.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:16.806 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:16.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:16.807 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:16.474+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:17.006+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:17.006+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:17.006+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:17.006+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:16.805+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:16.805+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:16.805+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:16.806+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:17.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:17.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:17.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:17.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:17.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:17.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:17.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:17.488+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:18.098 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:17.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:18.098 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:17.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:18.098 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:17.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:18.098 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:17.999+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:18.099 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:17.819+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:18.099 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:17.819+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:18.099 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:17.819+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:18.099 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:17.819+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:18.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:18.097+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 6053 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:18.816 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:18.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:18.816 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:18.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:18.817 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:18.458+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:18.817 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:18.459+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:19.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:19.038+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:19.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:19.038+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:19.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:19.038+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:19.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:19.039+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:19.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:18.815+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:19.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:18.815+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:19.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:18.815+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:19.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:18.816+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:19.814 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:19.484+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:19.815 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:19.484+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:19.815 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:19.484+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:19.815 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:19.485+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:20.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:20.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:20.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:20.038+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:19.813+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:19.813+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:19.813+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:20.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:19.814+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:20.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:20.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:20.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:20.486+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:20.863+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:20.863+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:21.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:20.863+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:21.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:20.864+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:21.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:21.065+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:21.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:21.065+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:21.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:21.065+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:21.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:21.066+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:21.851 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:21.481+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:21.851 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:21.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:21.851 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:21.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:21.851 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:21.483+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:22.093+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:22.093+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:22.093+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:22.094+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:22.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:21.849+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:22.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:21.849+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:22.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:21.849+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:21.850+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:22.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:22.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:22.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:22.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:22.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:22.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:22.524+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:23.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:22.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:23.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:22.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:23.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:22.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:23.093 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:22.901+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:23.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:23.098+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 6146 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:23.091+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:23.091+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:23.091+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:23.092+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:23.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:23.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:23.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:23.505+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:24.068+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:24.068+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:24.068+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:24.069+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:23.917+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:24.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:23.917+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:24.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:23.917+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:24.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:23.918+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:24.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:24.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:24.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:24.499+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:25.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:25.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:25.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:25.079+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:24.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:24.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:25.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:24.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:25.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:24.961+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:25.512+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:25.512+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:25.512+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:25.513+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:26.030+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:26.031+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:26.031+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:26.031+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:26.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:25.944+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:26.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:25.944+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:26.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:25.944+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:26.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:25.945+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:26.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:26.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:26.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:26.469+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:27.046+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:27.046+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:27.046+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:27.047+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:27.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:26.968+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:27.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:26.968+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:27.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:26.968+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:27.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:26.969+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:27.440+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:27.440+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:27.440+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:27.441+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:28.225 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:28.098+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 6238 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:28.225 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:28.028+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:28.225 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:28.028+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:28.225 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:28.028+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:28.225 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:28.029+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:28.225 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:27.921+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:28.226 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:27.921+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:28.226 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:27.921+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:28.226 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:27.921+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:28.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:28.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:28.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:28.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:28.448+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:29.080 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:28.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:29.080 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:28.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:29.080 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:28.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:29.080 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:28.961+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:29.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:29.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:29.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:29.079+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:29.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:29.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:29.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:29.450+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:30.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:30.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:30.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:30.067+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:30.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:29.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:30.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:29.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:30.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:29.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:30.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:29.929+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:30.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:30.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:30.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:30.407+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:30.914+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:30.914+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:30.914+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:30.914+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:31.099+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:31.099+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:31.099+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:31.100+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:31.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:31.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:31.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:31.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:31.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:31.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:31.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:31.439+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:32.080 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:31.959+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:32.080 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:31.959+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:32.080 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:31.959+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:32.080 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:31.960+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:32.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:32.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:32.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:32.079+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:32.401+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:32.401+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:32.401+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:32.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:32.402+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:33.101 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:32.963+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:33.101 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:32.963+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:33.101 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:32.963+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:33.101 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:32.964+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:33.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:33.100+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 6332 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:33.124+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:33.124+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:33.124+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:33.124+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:33.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:33.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:33.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:33.396+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:34.150+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:34.150+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:34.150+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:34.151+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:33.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:33.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:33.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:33.961+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:34.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:34.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:34.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:34.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:34.431+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:35.161 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:34.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:35.161 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:34.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:35.161 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:34.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:35.161 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:34.974+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:35.426 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:35.159+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:35.426 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:35.159+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:35.426 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:35.159+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:35.427 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:35.160+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:35.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:35.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:35.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:35.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:35.426+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:36.116+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:36.117+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:36.117+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:36.118+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:35.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:35.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:35.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:35.940+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:36.412+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:36.412+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:36.412+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:36.413+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:37.073+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:37.073+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:37.073+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:37.074+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:36.892+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:36.892+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:36.892+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:36.893+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:37.452+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:37.452+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:37.452+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:37.452+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:38.229 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:38.101+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 6426 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:38.229 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:37.923+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:38.230 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:37.923+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:38.230 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:37.923+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:38.230 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:37.924+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:38.230 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:38.051+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:38.230 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:38.051+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:38.230 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:38.051+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:38.230 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:38.052+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:38.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:38.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:38.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:38.435+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:39.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:39.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:39.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:39.013+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:38.932+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:38.932+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:38.932+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:38.932+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:39.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:39.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:39.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:39.446+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:40.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:40.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:40.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:40.014+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:39.956+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:39.956+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:39.956+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:39.956+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:40.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:40.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:40.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:40.483+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:40.975+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:40.975+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:40.975+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:40.976+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:40.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:40.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:40.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:40.958+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:41.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:16.033393+0000 front 2024-02-10T06:16:16.033400+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:41.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:16.033376+0000 front 2024-02-10T06:16:16.033333+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:41.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:16.033360+0000 front 2024-02-10T06:16:16.033415+0000 (oldest deadline 2024-02-10T06:16:38.933276+0000) 2024-02-10T06:26:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:41.518+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:41.926+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:41.926+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:41.926+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:41.927+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:41.926+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:41.926+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:41.926+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:41.926+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:42.505+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:43.103 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:42.923+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:43.103 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:42.923+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:43.103 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:42.923+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:43.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:42.924+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:43.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:42.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:43.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:42.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:43.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:42.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:43.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:42.894+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:43.102+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 6519 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:43.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:43.502+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:43.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:43.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:43.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:43.940+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:43.876+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:43.876+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:43.876+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:43.877+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:44.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:44.502+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:44.959+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:44.959+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:44.959+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:44.959+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:44.892+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:44.892+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:44.892+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:44.893+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:45.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:45.470+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:45.934+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:18.219821+0000 front 2024-02-10T06:16:18.219864+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:45.934+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:18.219791+0000 front 2024-02-10T06:16:18.219843+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:45.934+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:18.219801+0000 front 2024-02-10T06:16:18.219833+0000 (oldest deadline 2024-02-10T06:16:42.919823+0000) 2024-02-10T06:26:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:45.935+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:45.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:45.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:45.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:46.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:45.925+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:46.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:46.442+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:46.927+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:46.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:46.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:46.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:46.941+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:47.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:47.426+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:48.230 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:48.103+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 6611 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:48.231 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:47.932+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:48.231 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:47.946+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:48.231 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:47.946+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:48.231 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:47.946+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:48.231 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:47.947+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:48.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:48.386+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:48.905+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:48.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 since back 2024-02-10T06:16:14.959370+0000 front 2024-02-10T06:16:14.959330+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:48.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-02-10T06:16:14.959269+0000 front 2024-02-10T06:16:14.959282+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:48.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 since back 2024-02-10T06:16:14.959379+0000 front 2024-02-10T06:16:14.959362+0000 (oldest deadline 2024-02-10T06:16:40.259314+0000) 2024-02-10T06:26:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:48.901+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:49.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:49.367+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:50.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:49.907+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:49.900+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:50.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:50.412+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:51.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:50.944+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:50.946+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:51.462+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:52.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:51.978+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:51.948+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:52.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:52.454+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:52.932+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:52.936+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:53.473 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:53.105+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 6705 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:53.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:53.472+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:53.901+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:53.947+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:54.738 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:54.481+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:54.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-10T06:26:54.736+0000 7f18acdff700 0 monclient(hunting): authenticate timed out after 300 2024-02-10T06:26:54.738 INFO:teuthology.orchestra.run.smithi089.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-10T06:26:55.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:54.885+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:54.915+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:55.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:26:55.649 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1402, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi089 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps'" 2024-02-10T06:26:55.899 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b9da24eefb31450e842527ff80037c76 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1402, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi089 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps'" 2024-02-10T06:26:55.902 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-10T06:26:55.960 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-10T06:26:55.960 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:26:55.960 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:26:55.974 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:55.525+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:55.991 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:26:55.992 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:26:56.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-10T06:26:56.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:55.852+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:55.899+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:56.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:56.491+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:57.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:56.848+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:56.905+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:57.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:57.513+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:58.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:57.823+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:58.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:57.931+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:58.537 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:26:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:26:58.106+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 6798 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:26:58.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:58.781+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:58.536+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:58.886+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:59.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:26:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:26:59.804+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:26:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:26:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:26:59.582+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:00.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:26:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:26:59.885+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:00.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:00.599+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:00.781+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:01.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:00.854+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:01.598+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:01.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:01.797+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:01.898+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:02.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:02.572+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:03.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:02.827+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:02.894+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:03.523 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:03.107+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 6892 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:03.853 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:03.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:03.853 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:03.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:03.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:03.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:03.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:03.523+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:03.852+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:04.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:03.853+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:04.822 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:04.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:04.822 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:04.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:04.822 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:04.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:04.822 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:04.520+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:05.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:04.829+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:04.821+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:05.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:05.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:05.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:05.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:05.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:05.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:05.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:05.477+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:06.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:05.866+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:05.847+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:06.806 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:06.478+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:06.806 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:06.478+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:06.806 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:06.478+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:06.806 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:06.479+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:07.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:06.805+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:06.899+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:07.829 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:07.506+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:07.829 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:07.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:07.829 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:07.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:07.829 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:07.507+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:08.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:07.901+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:07.827+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:08.543 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:08.108+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 6985 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:08.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:08.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:08.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:08.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:08.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:08.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:08.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:08.543+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:09.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:08.821+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:08.938+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:09.833 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:09.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:09.834 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:09.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:09.834 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:09.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:09.834 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:09.521+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:09.832+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:09.955+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:09.955+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:09.955+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:09.956+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:10.799 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:10.798+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:10.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:10.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:10.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:10.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:10.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:10.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:10.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:10.474+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:10.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:10.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:11.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:10.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:11.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:10.950+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:11.740 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:11.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:11.740 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:11.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:11.741 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:11.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:11.741 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:11.471+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:11.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:11.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:11.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:11.929+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:12.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:11.788+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:12.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:11.789+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:12.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:11.789+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:12.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:11.789+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:12.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:12.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:12.789 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:12.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:12.789 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:12.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:12.789 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:12.432+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:13.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:12.897+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:13.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:12.897+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:13.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:12.897+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:13.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:12.898+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:12.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:12.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:12.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:12.787+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:13.469 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:13.107+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 7080 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:13.837 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:13.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:13.838 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:13.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:13.838 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:13.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:13.838 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:13.468+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:13.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:13.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:13.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:13.836+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:14.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:13.934+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:13.934+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:13.934+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:13.935+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:14.812 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:14.420+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:14.812 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:14.420+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:14.812 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:14.420+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:14.813 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:14.421+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:14.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:14.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:14.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:14.970+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:14.871+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:15.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:14.871+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:15.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:14.871+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:15.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:14.872+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:15.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:15.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:15.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:15.405+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:15.881+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:15.881+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:15.881+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:15.883+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:16.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:16.016+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:16.016+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:16.016+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:16.017+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:16.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:16.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:16.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:16.430+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:16.898+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:16.898+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:16.898+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:16.899+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:17.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:17.065+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:17.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:17.065+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:17.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:17.065+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:17.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:17.066+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:17.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:17.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:17.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:17.432+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:18.233 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:18.110+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 7174 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:18.234 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:18.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:18.234 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:18.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:18.234 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:18.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:18.234 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:18.091+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:18.235 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:17.905+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:18.235 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:17.905+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:18.235 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:17.905+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:18.235 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:17.906+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:18.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:18.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:18.398+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:18.399+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:19.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:19.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:19.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:19.114+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:19.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:18.926+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:19.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:18.926+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:19.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:18.926+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:19.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:18.926+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:19.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:19.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:19.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:19.442+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:20.075+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:20.075+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:20.075+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:20.076+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:20.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:19.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:20.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:19.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:20.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:19.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:20.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:19.946+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:20.466+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:20.466+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:20.466+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:20.467+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:21.038+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:21.038+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:21.039+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:21.039+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:20.925+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:21.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:20.925+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:21.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:20.925+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:21.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:20.925+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:21.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:21.426+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:21.426+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:21.426+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:21.426+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:21.994+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:21.994+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:21.994+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:21.995+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:22.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:21.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:22.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:21.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:22.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:21.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:22.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:21.879+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:22.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:22.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:22.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:22.403+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:22.972+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:22.972+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:22.972+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:22.973+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:23.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:22.915+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:23.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:22.915+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:23.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:22.915+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:23.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:22.916+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:23.385 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:23.111+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 7268 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:23.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:23.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:23.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:23.384+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:23.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:23.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:23.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:23.940+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:23.992+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:23.992+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:23.992+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:23.993+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:24.356+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:24.356+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:24.356+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:24.357+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:25.350 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:24.997+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:25.351 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:24.997+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:25.351 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:24.997+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:25.351 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:24.997+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:25.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:24.903+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:25.352 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:24.903+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:25.352 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:24.903+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:25.352 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:24.903+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:25.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:25.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:25.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:25.350+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:25.907+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:25.907+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:25.907+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:25.908+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:26.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:25.958+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:26.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:25.958+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:26.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:25.958+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:26.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:25.959+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:26.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:26.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:26.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:26.370+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:26.950+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:26.950+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:26.950+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:26.951+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:26.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:27.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:26.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:27.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:26.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:27.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:26.921+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:27.409+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:27.409+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:27.409+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:27.409+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:28.237 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:28.112+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 7361 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:28.237 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:27.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:28.238 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:27.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:28.238 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:27.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:28.238 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:27.937+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:28.238 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:27.921+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:28.238 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:27.921+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:28.238 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:27.921+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:28.238 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:27.922+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:28.423+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:28.423+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:28.423+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:28.424+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:28.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:28.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:28.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:28.929+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:29.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:28.959+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:29.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:28.959+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:29.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:28.959+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:29.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:28.960+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:29.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:29.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:29.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:29.448+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:29.989+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:29.989+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:29.989+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:29.989+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:30.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:29.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:30.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:29.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:30.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:29.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:30.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:29.961+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:30.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:30.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:30.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:30.411+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:30.988+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:30.988+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:30.988+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:30.989+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:31.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:30.912+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:31.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:30.912+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:31.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:30.912+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:31.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:30.912+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:31.665 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:31.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:31.666 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:31.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:31.666 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:31.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:31.666 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:31.411+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:31.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:31.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:31.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:31.941+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:32.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:31.944+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:32.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:31.944+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:32.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:31.944+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:32.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:31.945+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:32.364+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:32.364+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:32.364+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:32.364+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:32.906+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:32.906+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:32.906+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:32.907+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:33.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:32.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:33.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:32.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:33.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:32.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:33.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:32.971+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:33.559 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:33.112+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 7456 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:33.559 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:33.318+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:33.560 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:33.318+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:33.560 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:33.318+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:33.560 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:33.319+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:34.016+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:34.016+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:34.016+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:34.017+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:34.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:33.905+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:34.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:33.905+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:34.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:33.905+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:34.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:33.906+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:34.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:34.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:34.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:34.362+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:34.939+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:34.939+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:34.939+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:34.940+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:34.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:34.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:34.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:34.970+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:35.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:35.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:35.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:35.404+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:36.316 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:35.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:36.316 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:35.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:36.316 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:35.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:36.317 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:35.929+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:36.317 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:35.966+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:36.317 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:35.966+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:36.317 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:35.966+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:36.317 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:35.967+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:36.363+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:36.363+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:36.363+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:36.363+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:37.319 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:36.938+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:37.319 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:36.938+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:37.320 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:36.938+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:37.320 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:36.939+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:37.320 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:36.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:37.320 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:36.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:37.320 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:36.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:37.320 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:36.925+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:37.318+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:37.318+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:37.318+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:37.319+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:38.239 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:38.113+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 7550 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:38.239 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:37.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:38.239 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:37.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:38.239 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:37.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:38.240 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:37.977+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:38.240 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:37.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:38.240 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:37.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:38.240 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:37.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:38.240 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:37.921+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:38.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:38.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:38.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:38.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:38.280+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:39.280 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:38.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:39.280 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:38.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:39.280 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:38.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:39.281 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:38.999+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:39.281 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:38.965+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:39.281 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:38.965+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:39.281 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:38.965+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:39.282 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:38.966+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:39.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:39.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:39.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:39.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:39.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:39.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:39.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:39.280+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:40.314 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:40.033+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:40.314 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:40.033+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:40.314 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:40.033+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:40.314 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:40.033+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:40.315 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:40.013+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:40.315 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:40.013+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:40.315 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:40.013+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:40.315 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:40.014+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:40.313+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:40.313+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:40.313+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:40.313+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:41.278 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:41.080+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:41.278 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:41.080+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:41.278 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:41.080+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:41.279 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:41.081+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:41.279 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:40.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:41.279 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:40.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:41.279 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:40.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:41.279 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:40.984+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:41.276+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:41.276+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:41.276+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:41.277+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:42.315 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:42.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:42.315 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:42.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:42.315 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:42.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:42.315 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:42.096+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:42.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:41.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:42.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:41.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:42.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:41.979+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:42.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:41.980+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:42.314+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:42.314+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:42.314+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:42.315+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:43.007+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:43.007+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:43.007+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:43.008+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:43.560 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:43.115+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 7644 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:43.561 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:43.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:43.561 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:43.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:43.561 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:43.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:43.562 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:43.328+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:43.562 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:43.141+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:43.562 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:43.141+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:43.563 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:43.141+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:43.563 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:43.142+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:44.313 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:44.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:44.313 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:44.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:44.313 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:44.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:44.314 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:44.167+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:44.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:44.014+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:44.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:44.014+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:44.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:44.014+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:44.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:44.014+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:44.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:44.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:44.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:44.313+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:45.346 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:45.156+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:45.346 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:45.156+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:45.346 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:45.156+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:45.346 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:45.157+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:45.347 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:44.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:45.347 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:44.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:45.347 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:44.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:45.347 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:44.986+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:45.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:45.344+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:45.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:45.344+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:45.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:45.344+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:45.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:45.345+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:46.350 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:46.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:46.350 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:46.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:46.350 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:46.155+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:46.350 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:46.156+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:46.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:45.971+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:46.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:45.971+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:46.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:45.971+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:46.351 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:45.971+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:46.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:46.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:46.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:46.349+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:47.338 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:46.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:47.339 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:46.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:47.339 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:46.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:47.339 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:46.935+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:47.339 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:47.163+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:47.339 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:47.163+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:47.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:47.163+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:47.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:47.165+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:47.337+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:47.337+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:47.337+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:47.338+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:48.239 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:48.116+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 7737 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:48.240 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:48.168+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:48.240 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:48.168+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:48.240 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:48.168+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:48.240 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:48.168+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:48.241 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:47.951+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:48.241 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:47.951+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:48.241 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:47.951+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:48.241 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:47.952+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:48.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:48.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:48.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:48.360+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:49.346 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:49.137+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:49.347 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:49.137+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:49.347 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:49.137+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:49.347 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:49.137+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:49.348 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:48.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:49.348 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:48.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:49.348 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:48.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:49.348 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:48.921+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:49.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:49.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:49.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:49.346+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:50.134 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:49.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:50.134 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:49.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:50.134 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:49.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:50.134 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:49.879+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:50.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:50.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:50.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:50.340+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:50.132+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:50.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:50.132+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:50.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:50.132+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:50.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:50.134+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:51.312 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:51.111+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:51.313 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:51.111+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:51.313 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:51.111+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:51.313 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:51.112+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:51.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:50.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:51.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:50.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:51.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:50.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:51.314 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:50.898+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:51.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:51.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:51.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:51.312+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:52.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:51.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:52.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:51.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:52.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:51.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:52.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:51.901+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:52.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:52.328+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:52.328+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:52.328+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:52.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:52.114+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:52.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:52.114+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:52.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:52.114+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:52.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:52.114+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:53.075 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:52.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:53.075 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:52.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:53.075 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:52.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:53.075 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:52.946+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:53.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:53.117+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 7832 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:53.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:53.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:53.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:53.074+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:53.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:53.288+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:53.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:53.288+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:53.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:53.288+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:53.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:53.289+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:54.246 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:54.114+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:54.246 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:54.114+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:54.246 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:54.114+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:54.246 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:54.115+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:54.247 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:53.936+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:54.247 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:53.936+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:54.247 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:53.936+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:54.247 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:53.936+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:54.245+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:54.245+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:54.245+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:54.246+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:54.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:54.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:54.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:54.892+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:55.126+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:55.126+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:55.126+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:55.127+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:55.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:55.228+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:55.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:55.228+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:55.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:55.228+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:55.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:55.228+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:55.848+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:55.848+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:55.848+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:55.849+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:56.111+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:56.111+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:56.111+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:56.112+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:56.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:56.187+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:56.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:56.187+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:56.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:56.187+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:56.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:56.188+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:57.099 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:56.799+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:57.099 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:56.799+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:57.099 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:56.799+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:57.099 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:56.800+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:57.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:57.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:57.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:57.097+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:57.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:57.145+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:57.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:57.145+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:57.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:57.145+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:57.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:57.145+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:58.083 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:57.775+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:58.083 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:57.775+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:58.083 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:57.775+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:58.084 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:57.775+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:58.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:27:58.117+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 7925 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:27:58.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:58.113+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:58.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:58.113+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:58.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:58.113+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:58.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:58.114+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:58.081+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:58.081+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:58.081+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:58.082+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:59.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:59.044+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:59.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:59.044+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:59.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:59.044+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:27:59.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:27:59.045+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:58.824+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:58.824+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:58.824+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:27:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:58.825+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:27:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:59.147+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:59.147+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:59.147+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:27:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:27:59.148+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:00.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:00.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:00.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:00.058+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:59.821+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:59.821+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:59.821+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:27:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:27:59.821+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:00.111+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:00.111+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:00.111+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:00.112+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:01.153 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:01.070+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:01.154 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:01.070+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:01.154 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:01.070+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:01.154 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:01.071+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:01.154 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:00.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:01.155 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:00.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:01.155 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:00.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:01.155 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:00.865+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:01.152+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:01.152+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:01.152+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:01.153+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:02.076 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:01.888+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:02.076 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:01.888+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:02.077 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:01.888+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:02.077 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:01.889+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:02.167+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:02.167+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:02.167+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:02.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:02.168+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:02.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:02.075+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:02.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:02.075+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:02.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:02.075+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:02.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:02.076+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:03.048+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:03.048+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:03.048+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:03.049+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:02.849+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:03.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:02.849+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:03.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:02.849+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:03.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:02.849+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:03.562 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:03.119+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8018 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:03.562 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:03.164+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:03.562 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:03.164+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:03.562 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:03.164+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:03.562 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:03.165+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:04.206 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:04.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:04.207 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:04.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:04.207 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:04.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:04.207 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:04.039+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:04.207 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:03.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:04.207 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:03.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:04.207 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:03.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:04.207 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:03.892+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:04.205+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:04.205+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:04.205+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:04.206+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:05.204 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:05.007+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:05.204 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:05.007+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:05.204 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:05.007+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:05.204 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:05.008+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:05.205 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:04.895+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:05.205 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:04.895+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:05.205 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:04.895+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:05.205 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:04.896+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:05.202+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:05.202+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:05.202+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:05.203+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:06.165 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:06.030+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:06.165 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:06.030+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:06.166 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:06.030+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:06.166 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:06.031+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:06.166 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:05.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:06.166 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:05.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:06.166 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:05.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:06.166 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:05.894+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:06.164+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:06.164+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:06.164+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:06.165+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:07.185 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:06.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:07.186 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:06.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:07.186 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:06.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:07.186 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:06.997+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:07.186 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:06.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:07.186 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:06.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:07.186 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:06.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:07.187 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:06.881+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:07.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:07.184+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:07.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:07.184+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:07.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:07.184+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:07.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:07.185+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:08.147 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:08.120+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8110 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:08.148 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:08.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:08.148 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:08.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:08.148 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:08.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:08.148 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:08.041+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:08.149 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:07.889+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:08.149 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:07.889+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:08.149 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:07.889+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:08.149 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:07.890+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:08.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:08.155+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:08.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:08.155+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:08.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:08.155+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:08.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:08.156+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:09.145 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:08.883+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:09.145 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:08.883+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:09.145 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:08.883+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:09.145 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:08.884+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:09.146 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:09.034+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:09.146 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:09.034+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:09.146 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:09.034+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:09.146 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:09.035+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:09.144+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:09.144+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:09.144+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:09.145+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:10.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:10.170+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:10.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:10.170+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:10.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:10.170+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:10.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:10.171+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:10.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:10.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:10.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:10.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:10.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:10.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:10.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:10.065+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:10.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:09.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:10.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:09.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:10.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:09.928+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:10.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:09.929+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:11.195+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:11.195+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:11.195+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:11.196+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:11.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:11.089+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:11.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:11.089+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:11.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:11.089+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:11.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:11.089+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:11.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:10.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:11.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:10.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:11.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:10.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:11.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:10.958+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:12.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:12.164+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:12.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:12.164+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:12.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:12.164+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:12.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:12.164+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:12.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:12.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:12.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:12.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:12.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:12.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:12.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:12.065+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:12.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:11.925+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:12.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:11.925+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:12.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:11.925+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:12.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:11.927+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:13.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:12.914+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:13.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:12.914+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:13.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:12.914+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:13.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:12.915+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:13.562 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:13.121+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8204 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:13.562 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:13.206+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:13.563 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:13.206+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:13.563 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:13.206+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:13.563 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:13.207+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:13.563 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:13.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:13.563 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:13.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:13.563 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:13.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:13.563 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:13.107+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:14.250 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:14.129+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:14.250 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:14.129+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:14.250 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:14.129+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:14.250 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:14.130+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:14.251 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:13.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:14.251 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:13.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:14.251 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:13.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:14.251 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:13.935+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:14.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:14.249+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:14.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:14.249+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:14.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:14.249+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:14.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:14.249+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:15.299 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:15.092+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:15.299 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:15.092+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:15.299 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:15.092+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:15.299 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:15.092+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:15.299 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:14.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:15.300 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:14.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:15.300 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:14.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:15.300 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:14.984+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:15.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:15.297+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:15.297+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:15.298+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:15.298+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:16.328 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:16.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:16.328 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:16.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:16.328 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:16.140+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:16.328 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:16.141+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:16.329 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:16.007+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:16.329 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:16.007+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:16.329 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:16.007+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:16.329 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:16.008+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:16.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:16.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:16.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:16.327+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:17.288 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:17.123+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:17.288 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:17.123+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:17.288 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:17.123+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:17.289 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:17.124+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:17.289 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:17.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:17.289 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:17.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:17.289 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:17.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:17.289 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:17.019+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:17.287+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:17.287+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:17.287+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:17.287+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:18.246 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:18.122+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8298 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:18.246 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:18.110+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:18.246 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:18.110+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:18.246 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:18.110+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:18.247 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:18.111+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:18.247 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:17.972+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:18.247 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:17.972+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:18.247 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:17.972+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:18.247 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:17.972+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:18.274+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:18.274+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:18.274+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:18.275+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:19.322 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:19.158+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:19.323 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:19.158+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:19.323 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:19.158+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:19.323 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:19.159+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:19.323 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:18.987+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:19.324 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:18.987+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:19.324 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:18.987+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:19.324 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:18.989+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:19.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:19.321+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:19.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:19.321+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:19.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:19.321+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:19.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:19.322+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:20.276 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:20.128+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:20.277 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:20.128+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:20.277 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:20.128+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:20.277 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:20.129+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:20.277 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:20.007+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:20.277 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:20.008+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:20.277 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:20.008+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:20.277 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:20.008+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:20.275+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:20.275+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:20.275+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:20.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:20.276+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:21.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:21.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:21.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:21.029+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:21.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:21.266+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:21.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:21.266+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:21.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:21.266+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:21.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:21.267+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:21.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:21.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:21.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:21.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:21.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:21.166+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:21.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:21.167+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:22.296 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:22.043+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:22.296 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:22.043+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:22.297 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:22.043+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:22.297 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:22.043+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:22.297 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:22.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:22.297 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:22.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:22.297 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:22.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:22.298 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:22.191+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:22.295+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:22.295+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:22.295+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:22.296+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:23.343 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:23.123+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8391 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:23.344 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:23.154+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:23.344 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:23.154+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:23.344 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:23.154+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:23.344 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:23.155+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:23.344 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:23.058+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:23.345 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:23.058+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:23.345 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:23.058+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:23.345 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:23.059+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:23.342+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:23.342+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:23.342+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:23.343+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:24.193+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:24.193+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:24.193+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:24.194+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:24.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:24.089+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:24.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:24.089+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:24.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:24.089+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:24.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:24.089+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:24.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:24.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:24.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:24.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:24.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:24.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:24.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:24.311+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:25.345 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:25.146+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:25.345 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:25.146+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:25.345 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:25.146+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:25.346 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:25.147+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:25.346 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:25.071+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:25.346 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:25.071+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:25.346 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:25.071+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:25.346 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:25.072+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:25.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:25.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:25.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:25.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:25.344+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:26.181+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:26.181+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:26.181+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:26.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:26.181+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:26.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:26.050+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:26.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:26.050+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:26.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:26.050+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:26.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:26.051+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:26.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:26.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:26.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:26.375+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:27.339 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:27.177+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:27.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:27.177+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:27.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:27.177+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:27.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:27.178+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:27.340 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:27.053+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:27.340 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:27.053+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:27.340 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:27.053+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:27.341 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:27.053+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:27.338+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:27.338+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:27.338+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:27.339+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:28.359 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:28.124+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8483 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:28.396 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:28.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:28.396 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:28.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:28.396 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:28.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:28.396 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:28.079+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:28.397 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:28.173+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:28.397 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:28.173+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:28.398 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:28.173+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:28.398 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:28.174+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:28.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:28.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:28.358+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:28.359+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:29.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:29.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:29.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:29.104+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:29.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:29.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:29.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:29.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:29.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:29.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:29.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:29.193+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:29.355+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:29.355+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:29.355+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:29.356+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:30.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:30.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:30.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:30.317+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:30.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:30.225+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:30.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:30.225+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:30.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:30.225+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:30.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:30.225+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:30.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:30.122+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:30.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:30.122+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:30.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:30.122+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:30.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:30.123+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:31.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:31.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:31.184+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:31.185+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:31.090+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:31.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:31.090+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:31.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:31.090+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:31.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:31.091+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:31.669 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:31.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:31.669 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:31.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:31.669 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:31.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:31.669 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:31.366+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:32.196+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:32.196+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:32.196+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:32.196+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:32.075+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:32.075+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:32.075+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:32.075+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:32.391+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:32.391+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:32.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:32.391+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:32.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:32.392+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:33.041+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:33.041+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:33.041+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:33.042+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:33.415 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:33.125+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8577 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:33.415 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:33.215+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:33.415 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:33.215+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:33.416 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:33.215+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:33.416 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:33.216+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:33.413+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:33.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:33.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:33.415+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:34.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:34.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:34.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:34.193+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:33.994+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:34.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:33.994+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:34.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:33.994+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:34.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:33.995+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:34.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:34.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:34.450+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:34.451+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:34.967+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:34.967+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:34.967+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:34.968+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:35.174+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:35.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:35.174+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:35.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:35.174+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:35.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:35.175+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:35.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:35.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:35.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:35.446+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:36.004+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:36.004+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:36.004+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:36.004+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:36.133+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:36.133+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:36.133+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:36.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:36.133+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:36.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:36.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:36.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:36.468+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:36.991+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:36.991+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:36.991+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:36.992+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:37.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:37.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:37.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:37.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:37.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:37.097+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:37.471+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:37.471+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:37.471+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:37.472+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:38.252 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:38.127+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8671 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:38.252 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:37.951+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:38.253 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:37.951+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:38.253 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:37.951+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:38.253 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:37.952+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:38.253 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:38.122+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:38.253 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:38.122+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:38.253 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:38.122+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:38.253 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:38.122+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:38.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:38.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:38.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:38.500+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:39.138+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:39.138+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:39.138+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:39.139+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:38.969+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:38.969+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:38.969+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:38.970+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:39.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:39.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:39.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:39.478+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:40.162+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:40.162+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:40.162+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:40.162+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:39.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:40.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:39.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:40.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:39.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:40.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:39.960+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:40.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:40.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:40.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:40.503+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:41.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:41.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:41.192+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:41.193+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:41.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:40.998+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:41.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:40.998+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:41.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:40.998+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:41.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:40.998+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:41.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:41.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:41.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:41.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:41.490+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:42.226+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:42.226+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:42.226+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:42.226+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:41.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:42.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:41.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:42.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:41.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:42.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:41.984+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:42.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:42.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:42.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:42.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:42.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:42.496+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:42.959+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:42.959+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:42.959+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:42.959+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:43.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:43.127+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8761 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:43.505 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:43.238+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:43.505 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:43.238+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:43.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:43.238+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:43.506 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:43.239+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:43.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:43.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:43.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:43.503+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:44.277 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:43.929+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:44.277 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:43.929+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:44.277 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:43.929+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:44.277 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:43.930+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:44.276+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:44.276+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:44.276+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:44.276+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:44.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:44.488+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:44.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:44.488+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:44.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:44.488+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:44.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:44.488+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:45.275 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:44.936+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:45.275 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:44.936+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:45.275 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:44.936+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:45.275 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:44.936+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:45.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:45.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:45.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:45.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:45.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:45.503+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:45.273+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:45.273+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:45.273+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:45.273+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:46.311 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:45.918+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:46.311 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:45.918+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:46.312 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:45.918+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:46.312 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:45.919+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:46.466+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:46.466+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:46.466+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:46.467+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:46.310+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:46.310+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:46.310+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:46.311+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:47.329 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:46.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:47.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:46.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:47.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:46.900+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:47.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:46.901+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:47.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:47.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:47.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:47.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:47.509+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:47.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:47.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:47.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:47.329+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:48.130 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:47.862+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:48.130 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:47.862+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:48.131 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:47.862+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:48.131 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:47.863+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:48.488 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:48.129+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8846 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:48.489 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:48.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:48.489 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:48.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:48.489 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:48.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:48.489 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:48.305+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:48.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:48.877 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:48.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:48.877 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:48.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:48.877 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:48.488+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:49.329 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:48.860+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:49.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:48.860+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:49.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:48.860+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:49.330 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:48.861+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:49.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:49.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:49.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:49.523+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:49.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:49.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:49.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:49.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:49.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:49.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:49.329+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:50.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:49.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:50.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:49.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:50.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:49.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:50.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:49.843+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:50.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:50.282+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:50.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:50.282+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:50.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:50.282+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:50.570 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:50.283+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:50.834 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:50.568+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:50.834 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:50.568+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:50.834 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:50.568+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:50.834 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:50.569+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:50.833+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:50.833+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:50.833+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:50.834+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:51.558 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:51.251+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:51.559 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:51.251+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:51.559 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:51.251+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:51.559 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:51.252+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:51.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:51.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:51.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:51.787+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:51.557+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:51.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:51.557+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:51.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:51.557+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:51.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:51.558+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:52.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:52.208+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:52.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:52.208+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:52.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:52.208+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:52.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:52.209+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:52.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:52.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:52.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:52.602+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:52.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:52.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:52.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:52.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:52.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:52.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:52.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:52.739+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:53.560 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:53.130+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 8932 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:53.561 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:53.253+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:53.561 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:53.253+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:53.561 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:53.253+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:53.561 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:53.254+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:53.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:53.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:53.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:53.560+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:53.726+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:53.726+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:53.726+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:53.727+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:54.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:54.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:54.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:54.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:54.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:54.522+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:54.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:54.294+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:54.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:54.294+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:54.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:54.294+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:54.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:54.295+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:54.730+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:54.730+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:54.730+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:54.731+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:55.556 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:55.286+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:55.556 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:55.286+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:55.557 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:55.286+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:55.557 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:55.287+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:55.555+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:55.555+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:55.555+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:55.556+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:55.686+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:55.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:55.686+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:55.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:55.686+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:55.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:55.687+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-10T06:28:56.123 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-02-10T06:28:56.123 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:28:56.123 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:28:56.140 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-10T06:28:56.140 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:28:56.197 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-10T06:28:56.571 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:56.256+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:56.571 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:56.256+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:56.571 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:56.256+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:56.571 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:56.257+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:56.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:56.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:56.569+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:56.570+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:56.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:56.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:56.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:56.670+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:57.542 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:57.252+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:57.543 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:57.252+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:57.543 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:57.252+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:57.543 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:57.252+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:57.655+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:57.655+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:57.655+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:57.655+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:57.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:57.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:57.541+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:57.542+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:58.574 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:28:58.131+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 9015 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:28:58.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:58.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:58.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:58.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:58.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:58.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:58.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:58.292+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:58.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:58.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:58.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:58.685+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:58.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:58.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:58.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:58.574+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:59.583 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:59.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:59.583 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:59.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:59.583 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:59.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:28:59.583 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:28:59.284+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:59.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:59.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:59.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:28:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:28:59.582+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:28:59.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:59.716+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:59.716+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:59.716+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:28:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:28:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:28:59.717+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:00.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:00.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:00.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:00.534+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:00.310+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:00.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:00.310+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:00.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:00.310+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:00.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:00.311+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:01.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:00.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:01.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:00.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:01.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:00.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:01.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:00.706+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:01.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:01.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:01.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:01.521+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:01.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:01.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:01.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:01.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:01.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:01.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:01.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:01.341+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:01.693+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:01.693+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:01.693+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:01.694+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:02.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:02.496+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:02.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:02.496+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:02.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:02.496+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:02.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:02.498+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:02.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:02.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:02.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:02.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:02.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:02.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:02.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:02.384+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:02.677+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:02.677+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:02.677+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:02.677+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:03.426 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:03.132+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 9100 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:03.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:03.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:03.487+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:03.488+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:03.425+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:03.425+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:03.425+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:03.426+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:03.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:03.666+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:03.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:03.666+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:03.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:03.666+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:03.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:03.667+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:04.710 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:04.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:04.711 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:04.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:04.711 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:04.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:04.711 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:04.505+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:04.711 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:04.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:04.711 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:04.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:04.712 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:04.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:04.712 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:04.379+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:04.709+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:04.709+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:04.709+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:04.710+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:05.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:05.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:05.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:05.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:05.491+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:05.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:05.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:05.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:05.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:05.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:05.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:05.331+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:05.682+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:05.682+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:05.682+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:05.683+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:06.690 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:06.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:06.691 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:06.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:06.691 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:06.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:06.691 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:06.508+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:06.691 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:06.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:06.691 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:06.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:06.691 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:06.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:06.692 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:06.371+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:07.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:06.689+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:07.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:06.689+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:07.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:06.689+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:07.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:06.690+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:07.701 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:07.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:07.701 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:07.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:07.701 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:07.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:07.701 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:07.484+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:07.702 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:07.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:07.702 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:07.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:07.702 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:07.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:07.702 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:07.404+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:07.699+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:07.699+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:07.699+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:07.700+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:08.404 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:08.134+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 9185 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:08.703 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:08.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:08.703 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:08.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:08.703 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:08.403+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:08.703 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:08.403+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:08.703 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:08.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:08.703 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:08.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:08.704 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:08.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:08.704 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:08.508+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:09.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:08.701+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:09.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:08.701+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:09.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:08.701+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:09.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:08.702+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:09.686 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:09.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:09.686 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:09.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:09.686 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:09.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:09.686 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:09.536+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:09.686 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:09.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:09.686 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:09.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:09.687 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:09.383+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:09.687 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:09.384+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:09.684+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:09.684+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:09.684+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:09.685+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:10.644 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:10.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:10.644 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:10.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:10.644 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:10.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:10.644 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:10.371+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:10.644 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:10.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:10.645 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:10.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:10.645 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:10.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:10.645 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:10.508+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:11.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:10.643+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:11.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:10.643+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:11.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:10.643+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:11.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:10.644+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:11.672 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:11.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:11.672 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:11.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:11.672 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:11.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:11.672 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:11.521+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:11.673 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:11.402+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:11.673 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:11.402+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:11.673 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:11.402+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:11.673 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:11.403+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:11.673 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:11.628+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:11.674 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:11.628+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:11.674 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:11.628+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:11.674 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:11.629+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:12.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:12.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:12.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:12.508+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:12.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:12.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:12.378+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:12.378+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:12.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:12.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:12.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:12.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:12.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:12.607+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:12.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:12.608+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:13.412 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:13.135+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 9280 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:13.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:13.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:13.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:13.412+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:13.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:13.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:13.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:13.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:13.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:13.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:13.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:13.546+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:13.651+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:13.651+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:13.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:13.651+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:13.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:13.652+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:14.658 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:14.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:14.659 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:14.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:14.659 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:14.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:14.659 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:14.572+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:14.659 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:14.390+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:14.659 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:14.390+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:14.660 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:14.390+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:14.660 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:14.391+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:14.657+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:14.657+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:14.657+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:14.658+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:15.707 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:15.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:15.707 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:15.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:15.707 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:15.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:15.707 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:15.602+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:15.708 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:15.420+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:15.708 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:15.420+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:15.708 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:15.420+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:15.708 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:15.421+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:16.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:15.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:16.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:15.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:16.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:15.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:16.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:15.707+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:16.740 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:16.400+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:16.741 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:16.400+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:16.741 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:16.400+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:16.741 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:16.401+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:16.741 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:16.583+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:16.742 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:16.583+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:16.742 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:16.583+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:16.742 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:16.584+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:17.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:16.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:17.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:16.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:17.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:16.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:17.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:16.740+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:17.693 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:17.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:17.693 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:17.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:17.693 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:17.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:17.693 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:17.385+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:17.693 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:17.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:17.694 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:17.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:17.694 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:17.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:17.694 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:17.630+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:18.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:17.764+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:17.764+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:17.764+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:17.765+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:18.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:18.136+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 9372 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:18.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:18.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:18.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:18.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:18.356+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:18.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:18.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:18.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:18.666+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:19.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:18.729+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:19.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:18.729+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:19.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:18.729+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:19.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:18.730+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:19.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:19.615 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:19.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:19.615 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:19.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:19.616 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:19.334+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:19.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:19.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:19.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:19.656+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:20.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:19.740+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:20.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:19.740+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:20.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:19.740+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:20.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:19.741+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:20.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:20.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:20.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:20.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:20.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:20.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:20.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:20.342+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:20.741+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:20.741+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:20.741+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:20.742+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:21.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:20.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:21.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:20.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:21.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:20.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:21.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:20.620+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:21.294+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:21.294+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:21.294+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:21.294+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:21.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:21.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:21.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:21.666+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:21.738+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:21.738+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:21.738+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:21.739+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:22.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:22.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:22.318+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:22.319+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:22.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:22.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:22.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:22.737+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:23.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:22.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:23.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:22.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:23.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:22.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:23.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:22.665+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:23.568 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:23.137+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 9466 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:23.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:23.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:23.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:23.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:23.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:23.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:23.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:23.366+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:23.837 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:23.701+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:23.837 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:23.701+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:23.837 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:23.701+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:23.837 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:23.702+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:23.838 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:23.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:23.838 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:23.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:23.838 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:23.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:23.838 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:23.626+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:24.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:24.609+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:24.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:24.609+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:24.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:24.609+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:24.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:24.610+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:24.746 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:24.361+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:24.746 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:24.361+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:24.746 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:24.361+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:24.746 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:24.361+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:25.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:24.744+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:25.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:24.744+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:25.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:24.744+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:25.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:24.745+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:25.738 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:25.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:25.739 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:25.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:25.739 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:25.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:25.739 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:25.635+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:25.739 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:25.397+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:25.739 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:25.397+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:25.739 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:25.397+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:25.740 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:25.398+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:25.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:25.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:25.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:25.737+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:26.678 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:26.356+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:26.678 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:26.356+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:26.679 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:26.356+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:26.679 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:26.357+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:26.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:26.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:26.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:26.677+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:26.729+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:27.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:26.729+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:27.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:26.729+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:27.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:26.729+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:27.687 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:27.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:27.688 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:27.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:27.688 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:27.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:27.688 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:27.359+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:27.686+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:27.686+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:27.686+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:27.686+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:28.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:27.722+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:28.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:27.722+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:28.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:27.722+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:28.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:27.723+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:28.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:28.137+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 9557 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:28.345+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:28.345+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:28.345+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:28.346+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:28.747+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:28.747+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:28.747+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:28.747+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:29.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:28.717+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:29.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:28.717+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:29.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:28.717+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:29.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:28.718+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:29.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:29.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:29.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:29.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:29.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:29.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:29.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:29.307+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:29.755+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:29.755+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:29.755+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:29.755+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:30.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:29.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:30.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:29.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:30.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:29.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:30.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:29.695+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:30.353+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:30.353+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:30.353+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:30.354+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:30.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:30.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:30.786+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:30.787+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:31.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:30.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:31.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:30.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:31.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:30.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:31.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:30.678+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:31.672 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:31.385+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:31.672 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:31.385+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:31.673 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:31.385+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:31.673 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:31.386+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:31.673 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:31.631+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:31.673 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:31.631+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:31.673 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:31.631+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:31.673 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:31.632+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:31.812+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:31.812+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:31.812+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:31.813+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:32.675 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:32.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:32.676 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:32.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:32.676 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:32.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:32.676 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:32.379+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:33.052 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:32.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:33.052 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:32.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:33.052 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:32.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:33.053 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:32.819+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:33.053 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:32.672+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:33.053 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:32.672+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:33.053 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:32.672+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:33.053 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:32.673+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:33.354 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:33.140+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 9652 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:33.696 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:33.397+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:33.696 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:33.397+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:33.696 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:33.397+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:33.696 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:33.398+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:34.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:33.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:34.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:33.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:34.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:33.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:34.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:33.695+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:34.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:33.811+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:34.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:33.811+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:34.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:33.811+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:34.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:33.812+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:34.841 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:34.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:34.841 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:34.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:34.842 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:34.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:34.842 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:34.433+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:34.842 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:34.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:34.842 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:34.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:34.842 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:34.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:34.843 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:34.657+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:34.843 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:34.840+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:34.843 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:34.840+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:34.843 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:34.840+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:34.841+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:35.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:35.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:35.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:35.694+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:35.457+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:35.457+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:35.457+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:35.458+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:36.124 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:35.871+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:36.124 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:35.871+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:36.124 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:35.871+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:36.124 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:35.872+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:36.670+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:36.670+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:36.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:36.670+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:36.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:36.671+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:36.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:36.509+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:36.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:36.509+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:36.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:36.509+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:36.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:36.509+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:36.884+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:36.884+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:36.884+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:36.885+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:37.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:37.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:37.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:37.635+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:37.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:37.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:37.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:37.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:37.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:37.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:37.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:37.509+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:38.142 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:37.855+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:38.142 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:37.855+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:38.143 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:37.855+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:38.143 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:37.856+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:38.553 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:38.141+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 9744 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:38.849 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:38.552+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:38.850 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:38.552+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:38.850 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:38.552+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:38.850 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:38.553+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:38.850 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:38.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:38.850 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:38.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:38.851 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:38.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:38.851 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:38.626+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:39.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:38.848+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:39.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:38.848+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:39.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:38.848+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:39.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:38.849+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:39.585+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:39.585+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:39.585+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:39.585+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:39.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:39.545+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:39.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:39.545+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:39.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:39.545+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:39.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:39.545+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:39.884+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:39.884+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:39.884+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:39.885+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:40.563+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:40.563+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:40.563+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:40.565+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:40.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:40.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:40.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:40.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:40.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:40.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:40.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:40.561+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:40.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:40.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:40.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:40.921+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:41.894 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:41.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:41.895 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:41.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:41.895 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:41.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:41.895 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:41.607+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:41.895 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:41.612+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:41.895 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:41.612+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:41.895 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:41.612+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:41.896 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:41.612+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:42.268 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:41.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:42.268 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:41.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:42.268 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:41.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:42.268 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:41.894+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:42.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:42.649+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:42.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:42.649+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:42.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:42.649+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:42.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:42.649+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:42.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:42.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:42.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:42.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:42.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:42.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:42.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:42.575+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:43.156 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:43.142+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 9837 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:43.156 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:42.895+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:43.157 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:42.895+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:43.157 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:42.895+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:43.157 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:42.896+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:43.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:43.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:43.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:43.574+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:43.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:43.618+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:43.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:43.618+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:43.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:43.618+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:43.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:43.619+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:43.867+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:43.867+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:43.867+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:43.868+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:44.590+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:44.590+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:44.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:44.590+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:44.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:44.591+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:44.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:44.583+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:44.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:44.583+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:44.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:44.583+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:44.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:44.584+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:45.340 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:44.883+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:45.340 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:44.883+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:45.340 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:44.883+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:45.341 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:44.884+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:45.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:45.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:45.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:45.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:45.582+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:45.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:45.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:45.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:45.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:45.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:45.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:45.608 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:45.547+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:45.927+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:45.927+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:45.927+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:45.929+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:46.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:46.628+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:46.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:46.628+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:46.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:46.628+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:46.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:46.629+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:46.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:46.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:46.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:46.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:46.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:46.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:46.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:46.509+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:46.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:46.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:46.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:46.888+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:47.852 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:47.613+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:47.852 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:47.613+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:47.853 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:47.613+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:47.853 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:47.613+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:47.853 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:47.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:47.853 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:47.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:47.853 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:47.529+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:47.853 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:47.529+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:47.850+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:47.850+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:47.850+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:47.852+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:48.565 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:48.143+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 9929 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:48.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:48.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:48.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:48.586+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:48.564+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:48.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:48.564+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:48.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:48.564+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:48.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:48.565+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:48.871+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:48.871+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:48.871+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:48.871+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:49.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:49.626+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:49.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:49.626+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:49.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:49.626+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:49.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:49.627+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:49.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:49.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:49.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:49.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:49.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:49.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:49.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:49.571+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:49.908+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:49.908+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:49.908+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:49.909+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:50.936 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:50.640+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:50.936 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:50.640+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:50.936 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:50.640+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:50.936 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:50.641+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:50.937 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:50.617+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:50.937 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:50.617+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:50.937 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:50.617+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:50.937 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:50.618+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:50.935+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:50.935+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:50.935+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:50.936+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:51.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:51.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:51.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:51.585+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:51.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:51.658+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:51.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:51.658+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:51.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:51.658+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:51.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:51.659+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:51.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:51.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:51.957+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:51.958+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:52.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:52.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:52.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:52.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:52.580+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:52.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:52.627+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:52.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:52.627+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:52.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:52.627+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:52.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:52.628+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:53.270 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:53.145+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 10025 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:53.271 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:52.992+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:53.271 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:52.992+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:53.271 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:52.992+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:53.271 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:52.993+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:53.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:53.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:53.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:53.590+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:53.588+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:53.588+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:53.588+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:53.589+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:54.014+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:54.014+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:54.014+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:54.015+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:54.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:54.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:54.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:54.630+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:54.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:54.604+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:54.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:54.604+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:54.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:54.604+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:54.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:54.605+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:55.038+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:55.038+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:55.038+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:55.038+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:55.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:55.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:55.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:55.626+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:55.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:55.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:55.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:55.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:55.585+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:56.013+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:56.013+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:56.013+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:56.013+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:56.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:56.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:56.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:56.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:56.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:56.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:56.602+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:56.576+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:56.576+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:56.576+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:56.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:56.577+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:57.049+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:57.049+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:57.049+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:57.050+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:58.030 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:57.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:58.030 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:57.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:58.030 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:57.639+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:58.030 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:57.640+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:58.031 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:57.610+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:58.031 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:57.610+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:58.031 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:57.610+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:58.031 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:57.611+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:58.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:29:58.146+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 10118 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:29:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:58.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:58.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:58.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:58.030+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:59.075 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:58.625+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:59.076 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:58.625+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:59.076 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:58.625+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:29:59.076 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:58.626+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:59.076 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:58.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:59.076 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:58.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:59.076 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:58.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:29:59.077 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:58.679+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:29:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:59.074+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:59.074+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:59.074+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:29:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:29:59.075+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:00.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:59.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:00.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:59.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:00.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:59.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:00.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:29:59.666+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:59.671+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:00.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:59.671+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:00.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:59.671+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:00.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:29:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:29:59.672+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:00.115+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:00.115+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:00.115+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:00.116+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:01.003 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:00.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:01.003 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:00.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:01.003 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:00.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:01.004 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:00.679+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:01.004 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:00.696+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:01.004 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:00.696+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:01.004 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:00.696+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:01.004 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:00.697+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:01.082+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:01.082+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:01.082+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:01.082+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:02.056 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:01.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:02.057 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:01.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:02.057 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:01.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:02.057 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:01.655+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:02.057 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:01.692+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:02.057 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:01.692+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:02.057 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:01.692+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:02.058 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:01.692+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:02.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:02.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:02.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:02.056+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:03.088 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:02.624+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:03.088 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:02.624+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:03.088 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:02.624+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:03.088 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:02.625+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:03.089 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:02.646+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:03.089 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:02.646+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:03.089 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:02.646+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:03.089 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:02.647+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:03.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:03.147+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 10212 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:03.086+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:03.086+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:03.086+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:03.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:03.087+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:03.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:03.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:03.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:03.615+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:03.652+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:03.652+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:03.652+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:03.653+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:04.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:04.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:04.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:04.130+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:04.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:04.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:04.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:04.606+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:05.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:04.624+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:05.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:04.624+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:05.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:04.624+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:05.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:04.625+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:05.573 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:05.115+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:05.573 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:05.115+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:05.573 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:05.115+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:05.573 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:05.116+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:05.571+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:05.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:05.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:05.573+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:05.648+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:05.648+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:05.648+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:05.648+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:06.073+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:06.073+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:06.073+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:06.074+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:06.556+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:06.556+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:06.556+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:06.556+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:06.632+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:06.632+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:06.632+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:06.632+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:07.519 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:07.113+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:07.520 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:07.113+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:07.520 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:07.113+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:07.520 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:07.114+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:07.518+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:07.518+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:07.518+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:07.519+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:07.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:07.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:07.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:07.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:07.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:07.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:07.586+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:08.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:08.148+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 10304 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:08.099+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:08.099+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:08.099+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:08.100+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:08.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:08.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:08.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:08.559+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:08.597+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:08.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:08.597+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:08.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:08.597+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:08.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:08.598+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:09.081+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:09.081+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:09.081+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:09.082+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:09.574+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:09.574+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:09.574+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:09.575+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:09.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:09.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:09.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:09.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:09.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:09.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:09.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:09.581+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:10.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:10.080+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:10.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:10.080+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:10.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:10.080+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:10.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:10.081+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:10.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:10.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:10.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:10.629+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:10.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:10.559+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:10.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:10.559+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:10.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:10.559+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:10.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:10.560+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:11.531 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:11.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:11.531 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:11.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:11.532 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:11.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:11.532 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:11.131+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:11.529+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:11.529+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:11.529+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:11.530+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:11.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:11.624+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:11.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:11.624+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:11.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:11.624+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:11.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:11.625+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:12.566 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:12.176+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:12.566 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:12.176+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:12.566 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:12.176+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:12.566 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:12.177+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:12.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:12.564+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:12.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:12.564+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:12.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:12.564+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:12.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:12.565+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:12.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:12.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:12.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:12.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:12.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:12.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:12.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:12.615+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:13.272 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:13.149+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 10400 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:13.272 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:13.205+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:13.272 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:13.205+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:13.273 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:13.205+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:13.273 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:13.205+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:13.602+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:13.602+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:13.602+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:13.602+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:13.541+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:13.541+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:13.541+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:13.542+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:14.534 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:14.197+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:14.534 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:14.197+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:14.534 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:14.197+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:14.534 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:14.197+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:14.564+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:14.564+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:14.564+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:14.565+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:14.532+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:14.532+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:14.532+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:14.534+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:15.502 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:15.194+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:15.503 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:15.194+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:15.503 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:15.194+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:15.503 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:15.195+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:15.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:15.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:15.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:15.605+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:15.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:15.501+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:15.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:15.501+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:15.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:15.501+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:15.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:15.502+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:16.240+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:16.240+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:16.240+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:16.241+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:16.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:16.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:16.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:16.638+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:16.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:16.529+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:16.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:16.529+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:16.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:16.529+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:16.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:16.529+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:17.290+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:17.290+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:17.290+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:17.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:17.291+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:17.523+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:17.523+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:17.523+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:17.523+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:18.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:17.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:18.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:17.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:18.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:17.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:18.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:17.641+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:18.541 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:18.151+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 10492 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:18.542 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:18.315+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:18.542 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:18.315+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:18.542 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:18.315+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:18.542 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:18.315+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:18.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:18.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:18.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:18.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:18.659+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:18.540+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:18.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:18.540+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:18.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:18.540+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:18.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:18.541+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:19.515 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:19.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:19.516 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:19.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:19.516 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:19.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:19.516 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:19.271+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:19.513+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:19.513+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:19.513+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:19.514+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:19.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:19.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:19.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:19.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:19.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:19.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:19.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:19.690+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:20.468 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:20.270+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:20.468 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:20.270+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:20.468 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:20.270+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:20.468 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:20.271+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:20.467+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:20.467+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:20.467+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:20.468+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:20.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:20.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:20.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:20.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:20.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:20.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:20.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:20.694+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:21.310+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:21.310+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:21.310+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:21.311+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:21.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:21.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:21.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:21.483+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:21.736+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:21.737+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:21.737+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:21.737+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:22.471 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:22.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:22.471 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:22.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:22.471 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:22.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:22.471 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:22.355+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:22.697+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:22.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:22.697+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:22.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:22.697+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:22.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:22.698+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:22.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:22.470+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:22.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:22.470+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:22.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:22.470+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:22.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:22.470+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:23.448 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:23.152+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 10586 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:23.449 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:23.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:23.449 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:23.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:23.449 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:23.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:23.449 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:23.323+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:23.731 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:23.447+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:23.731 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:23.447+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:23.732 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:23.447+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:23.732 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:23.449+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:23.730+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:23.730+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:23.730+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:23.731+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:24.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:24.316+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:24.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:24.316+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:24.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:24.316+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:24.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:24.317+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:24.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:24.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:24.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:24.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:24.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:24.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:24.448+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:25.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:24.776+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:25.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:24.776+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:25.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:24.776+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:25.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:24.776+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:25.318+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:25.318+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:25.318+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:25.318+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:25.477+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:25.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:25.477+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:25.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:25.477+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:25.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:25.478+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:26.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:25.745+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:26.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:25.745+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:26.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:25.745+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:26.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:25.746+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:26.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:26.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:26.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:26.431+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:26.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:26.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:26.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:26.332+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:26.727+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:26.727+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:26.727+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:26.728+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:27.770 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:27.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:27.771 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:27.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:27.771 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:27.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:27.771 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:27.439+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:27.771 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:27.377+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:27.771 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:27.377+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:27.771 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:27.377+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:27.772 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:27.378+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:27.769+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:27.769+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:27.769+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:27.770+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:28.424 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:28.153+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 10677 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:28.425 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:28.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:28.425 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:28.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:28.425 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:28.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:28.425 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:28.352+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:28.736 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:28.423+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:28.736 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:28.423+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:28.736 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:28.423+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:28.736 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:28.424+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:29.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:28.734+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:29.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:28.735+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:29.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:28.735+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:29.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:28.736+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:29.716 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:29.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:29.716 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:29.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:29.717 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:29.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:29.717 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:29.399+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:29.717 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:29.381+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:29.717 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:29.381+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:29.717 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:29.381+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:29.717 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:29.382+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:30.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:29.715+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:30.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:29.715+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:30.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:29.715+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:30.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:29.716+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:30.372+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:30.372+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:30.372+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:30.373+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:30.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:30.384+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:30.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:30.384+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:30.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:30.384+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:30.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:30.385+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:31.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:30.740+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:31.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:30.740+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:31.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:30.740+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:31.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:30.740+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:31.352+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:31.352+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:31.352+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:31.352+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:31.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:31.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:31.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:31.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:31.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:31.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:31.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:31.365+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:32.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:31.749+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:32.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:31.749+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:32.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:31.749+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:32.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:31.749+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:32.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:32.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:32.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:32.333+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:32.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:32.347+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:32.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:32.347+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:32.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:32.347+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:32.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:32.348+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:33.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:32.715+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:33.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:32.715+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:33.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:32.715+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:33.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:32.716+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:33.573 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:33.154+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 10773 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:33.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:33.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:33.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:33.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:33.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:33.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:33.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:33.310+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:33.575 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:33.338+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:33.575 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:33.338+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:33.575 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:33.338+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:33.575 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:33.339+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:33.752+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:33.752+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:33.752+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:33.752+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:34.314+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:34.314+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:34.314+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:34.314+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:34.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:34.368+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:34.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:34.368+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:34.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:34.368+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:34.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:34.369+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:35.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:34.708+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:35.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:34.708+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:35.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:34.708+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:35.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:34.709+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:35.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:35.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:35.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:35.344+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:35.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:35.282+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:35.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:35.282+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:35.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:35.282+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:35.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:35.283+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:36.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:35.675+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:36.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:35.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:36.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:35.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:36.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:35.676+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:36.302+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:36.302+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:36.302+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:36.303+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:36.341+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:36.341+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:36.341+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:36.341+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:37.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:36.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:37.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:36.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:37.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:36.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:37.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:36.664+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:37.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:37.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:37.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:37.350+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:37.328+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:37.328+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:37.328+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:37.329+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:38.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:37.702+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:38.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:37.702+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:38.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:37.702+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:38.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:37.702+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:38.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:38.156+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 10866 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:38.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:38.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:38.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:38.376+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:38.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:38.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:38.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:38.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:38.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:38.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:38.608 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:38.356+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:39.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:38.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:39.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:38.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:39.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:38.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:39.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:38.678+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:39.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:39.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:39.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:39.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:39.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:39.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:39.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:39.346+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:39.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:39.394+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:39.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:39.394+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:39.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:39.394+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:39.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:39.395+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:40.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:39.658+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:40.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:39.658+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:40.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:39.658+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:40.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:39.658+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:40.316+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:40.316+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:40.316+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:40.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:40.317+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:40.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:40.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:40.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:40.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:40.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:40.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:40.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:40.384+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:41.047 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:40.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:41.048 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:40.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:41.048 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:40.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:41.048 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:40.624+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:41.591 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:41.321+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:41.591 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:41.321+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:41.591 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:41.321+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:41.591 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:41.321+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:41.592 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:41.405+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:41.592 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:41.405+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:41.592 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:41.405+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:41.592 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:41.406+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:41.590+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:41.590+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:41.590+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:41.590+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:42.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:42.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:42.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:42.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:42.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:42.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:42.404+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:42.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:42.317+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:42.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:42.317+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:42.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:42.317+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:42.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:42.318+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:43.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:42.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:43.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:42.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:43.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:42.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:43.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:42.640+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:43.420 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:43.157+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 10960 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:43.420 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:43.353+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:43.420 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:43.353+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:43.420 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:43.353+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:43.421 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:43.354+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:43.708 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:43.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:43.708 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:43.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:43.708 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:43.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:43.709 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:43.668+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:43.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:43.418+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:43.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:43.418+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:43.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:43.418+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:43.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:43.420+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:44.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:44.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:44.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:44.332+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:44.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:44.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:44.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:44.419+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:44.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:44.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:44.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:44.679+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:45.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:45.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:45.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:45.340+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:45.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:45.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:45.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:45.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:45.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:45.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:45.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:45.438+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:46.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:45.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:45.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:45.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:45.697+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:46.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:46.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:46.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:46.331+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:46.394+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:46.394+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:46.394+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:46.395+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:47.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:46.727+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:47.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:46.727+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:47.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:46.727+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:47.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:46.728+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:47.684 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:47.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:47.685 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:47.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:47.685 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:47.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:47.685 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:47.371+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:47.685 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:47.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:47.685 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:47.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:47.685 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:47.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:47.686 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:47.390+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:47.683+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:47.683+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:47.683+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:47.684+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:48.421 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:48.158+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 11051 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:48.422 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:48.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:48.422 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:48.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:48.422 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:48.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:48.422 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:48.393+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:48.691 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:48.420+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:48.692 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:48.420+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:48.692 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:48.420+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:48.692 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:48.421+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:48.690+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:48.690+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:48.690+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:48.691+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:49.681 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:49.386+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:49.681 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:49.386+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:49.681 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:49.386+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:49.682 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:49.387+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:49.682 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:49.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:49.682 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:49.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:49.682 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:49.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:49.682 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:49.420+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:49.680+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:49.680+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:49.680+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:50.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:49.680+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:50.344+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:50.344+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:50.344+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:50.345+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:50.372+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:50.372+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:50.372+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:50.373+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:50.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:50.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:50.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:50.654+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:51.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:51.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:51.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:51.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:51.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:51.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:51.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:51.343+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:51.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:51.356+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:51.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:51.356+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:51.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:51.356+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:51.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:51.357+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:51.615+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:51.615+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:51.615+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:51.616+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:52.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:52.310+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:52.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:52.310+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:52.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:52.310+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:52.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:52.311+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:52.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:52.407+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:52.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:52.407+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:52.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:52.407+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:52.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:52.408+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:52.924 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:52.620+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:52.925 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:52.620+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:52.925 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:52.620+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:52.925 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:52.621+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:53.183 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:53.160+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 11146 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:53.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:53.408+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:53.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:53.408+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:53.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:53.408+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:53.576 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:53.409+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:53.576 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:53.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:53.576 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:53.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:53.576 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:53.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:53.576 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:53.346+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:53.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:53.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:53.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:53.582+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:54.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:54.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:54.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:54.559+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:54.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:54.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:54.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:54.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:54.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:54.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:54.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:54.322+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:54.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:54.393+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:54.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:54.393+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:54.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:54.393+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:54.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:54.394+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:55.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:55.436+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:55.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:55.436+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:55.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:55.436+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:55.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:55.437+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:55.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:55.327+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:55.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:55.327+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:55.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:55.328+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:55.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:55.328+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:55.598+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:55.598+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:55.598+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:55.599+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:56.270 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-10T06:30:56.270 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-02-10T06:30:56.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:30:56.271 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-02-10T06:30:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-02-10T06:30:56.312 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/examples: Cannot mkdir: Input/output error 2024-02-10T06:30:56.312 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/examples/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.312 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/examples/epoll-test.c: Cannot open: Input/output error 2024-02-10T06:30:56.312 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/src: Cannot mkdir: Input/output error 2024-02-10T06:30:56.312 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/src/epoll.c: Cannot open: Input/output error 2024-02-10T06:30:56.312 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/include: Cannot mkdir: Input/output error 2024-02-10T06:30:56.312 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/include/epoll.h: Cannot open: Input/output error 2024-02-10T06:30:56.313 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/include/dbllist.h: Cannot open: Input/output error 2024-02-10T06:30:56.313 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2: Cannot utime: Input/output error 2024-02-10T06:30:56.313 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/quotactl: Cannot mkdir: Input/output error 2024-02-10T06:30:56.313 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/quotactl/quotactl01.c: Cannot open: Input/output error 2024-02-10T06:30:56.313 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/quotactl/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.313 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate: Cannot mkdir: Input/output error 2024-02-10T06:30:56.313 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate/truncate04.c: Cannot open: Input/output error 2024-02-10T06:30:56.313 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate/truncate02.c: Cannot open: Input/output error 2024-02-10T06:30:56.313 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.314 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate/truncate03.c: Cannot open: Input/output error 2024-02-10T06:30:56.314 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate/truncate01.c: Cannot open: Input/output error 2024-02-10T06:30:56.314 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/profil: Cannot mkdir: Input/output error 2024-02-10T06:30:56.314 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/profil/profil01.c: Cannot open: Input/output error 2024-02-10T06:30:56.314 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/profil/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.314 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/timer_getoverrun: Cannot mkdir: Input/output error 2024-02-10T06:30:56.314 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/timer_getoverrun/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.314 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/timer_getoverrun/timer_getoverrun01.c: Cannot open: Input/output error 2024-02-10T06:30:56.315 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync: Cannot mkdir: Input/output error 2024-02-10T06:30:56.315 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/msync04.c: Cannot open: Input/output error 2024-02-10T06:30:56.315 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.315 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/msync01.c: Cannot open: Input/output error 2024-02-10T06:30:56.315 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/msync02.c: Cannot open: Input/output error 2024-02-10T06:30:56.315 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/msync03.c: Cannot open: Input/output error 2024-02-10T06:30:56.315 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/msync05.c: Cannot open: Input/output error 2024-02-10T06:30:56.315 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigprocmask: Cannot mkdir: Input/output error 2024-02-10T06:30:56.316 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigprocmask/sigprocmask01.c: Cannot open: Input/output error 2024-02-10T06:30:56.316 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigprocmask/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.316 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaltstack: Cannot mkdir: Input/output error 2024-02-10T06:30:56.316 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaltstack/sigaltstack02.c: Cannot open: Input/output error 2024-02-10T06:30:56.316 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaltstack/sigaltstack01.c: Cannot open: Input/output error 2024-02-10T06:30:56.316 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaltstack/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.316 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs: Cannot mkdir: Input/output error 2024-02-10T06:30:56.316 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs02.c: Cannot open: Input/output error 2024-02-10T06:30:56.317 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs05.c: Cannot open: Input/output error 2024-02-10T06:30:56.317 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.317 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs06.c: Cannot open: Input/output error 2024-02-10T06:30:56.317 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs03.c: Cannot open: Input/output error 2024-02-10T06:30:56.317 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs04.c: Cannot open: Input/output error 2024-02-10T06:30:56.317 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs01.c: Cannot open: Input/output error 2024-02-10T06:30:56.317 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/readlinkat: Cannot mkdir: Input/output error 2024-02-10T06:30:56.317 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/readlinkat/readlinkat01.c: Cannot open: Input/output error 2024-02-10T06:30:56.318 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/readlinkat/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.318 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/wait: Cannot mkdir: Input/output error 2024-02-10T06:30:56.318 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/wait/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.318 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/wait/wait02.c: Cannot open: Input/output error 2024-02-10T06:30:56.318 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall: Cannot mkdir: Input/output error 2024-02-10T06:30:56.318 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall/socketcall03.c: Cannot open: Input/output error 2024-02-10T06:30:56.319 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall/socketcall04.c: Cannot open: Input/output error 2024-02-10T06:30:56.319 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.319 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall/socketcall01.c: Cannot open: Input/output error 2024-02-10T06:30:56.319 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall/socketcall02.c: Cannot open: Input/output error 2024-02-10T06:30:56.319 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchownat: Cannot mkdir: Input/output error 2024-02-10T06:30:56.320 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchownat/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.320 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchownat/fchownat01.c: Cannot open: Input/output error 2024-02-10T06:30:56.320 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2: Cannot mkdir: Input/output error 2024-02-10T06:30:56.320 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/dup203.c: Cannot open: Input/output error 2024-02-10T06:30:56.320 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/dup204.c: Cannot open: Input/output error 2024-02-10T06:30:56.320 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/dup205.c: Cannot open: Input/output error 2024-02-10T06:30:56.320 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.322 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/dup201.c: Cannot open: Input/output error 2024-02-10T06:30:56.322 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/dup202.c: Cannot open: Input/output error 2024-02-10T06:30:56.322 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sbrk: Cannot mkdir: Input/output error 2024-02-10T06:30:56.322 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sbrk/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.322 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sbrk/sbrk01.c: Cannot open: Input/output error 2024-02-10T06:30:56.322 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unshare: Cannot mkdir: Input/output error 2024-02-10T06:30:56.322 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unshare/Makefile: Cannot open: Input/output error 2024-02-10T06:30:56.322 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unshare/unshare01.c: Cannot open: Input/output error 2024-02-10T06:30:56.323 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-02-10T06:30:56.323 INFO:tasks.workunit.client.0.smithi089.stderr:gzip: stdin: Stale file handle 2024-02-10T06:30:56.323 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unshare/unshare02.c: Cannot open: Input/output error 2024-02-10T06:30:56.323 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fadvise: Cannot mkdir: Input/output error 2024-02-10T06:30:56.323 INFO:tasks.workunit.client.0.smithi089.stderr:tar: Unexpected EOF in archive 2024-02-10T06:30:56.323 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls: Cannot utime: Input/output error 2024-02-10T06:30:56.323 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases/kernel: Cannot utime: Input/output error 2024-02-10T06:30:56.323 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-02-10T06:30:56.323 INFO:tasks.workunit.client.0.smithi089.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-02-10T06:30:56.323 INFO:tasks.workunit.client.0.smithi089.stderr:tar: Error is not recoverable: exiting now 2024-02-10T06:30:56.324 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-02-10T06:30:56.324 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-10T06:30:56.325 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-02-10T06:30:56.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-10T06:30:56.326 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-10T06:30:56.401 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-10T06:30:56.401 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-10T06:30:56.562 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:56.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:56.562 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:56.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:56.562 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:56.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:56.562 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:56.334+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:56.563 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:56.458+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:56.563 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:56.458+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:56.563 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:56.458+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:56.563 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:56.459+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:56.563 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:56.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:56.564 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:56.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:56.564 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:56.559+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:56.560+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:57.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:57.509+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:57.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:57.509+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:57.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:57.509+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:57.574 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:57.510+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:57.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:57.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:57.575 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:57.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:57.575 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:57.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:57.575 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:57.322+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:57.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:57.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:57.572+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:57.572+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:58.520 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:30:58.160+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 11239 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:30:58.520 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:58.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:58.520 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:58.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:58.520 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:58.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:58.521 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:58.358+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:58.518+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:58.518+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:58.518+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:58.519+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:58.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:58.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:58.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:58.535+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:59.550+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:59.550+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:59.550+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:30:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:30:59.550+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:59.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:59.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:59.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:59.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:59.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:59.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:30:59.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:30:59.574+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:30:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:59.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:59.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:59.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:59.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:59.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:30:59.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:30:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:30:59.366+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:00.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:00.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:00.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:00.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:00.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:00.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:00.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:00.407+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:00.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:00.550+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:00.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:00.550+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:00.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:00.550+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:00.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:00.551+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:00.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:00.597+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:00.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:00.597+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:00.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:00.597+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:00.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:00.598+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:01.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:01.457+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:01.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:01.457+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:01.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:01.457+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:01.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:01.458+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:01.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:01.561+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:01.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:01.561+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:01.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:01.561+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:01.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:01.562+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:01.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:01.578+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:01.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:01.578+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:01.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:01.578+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:01.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:01.579+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:02.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:02.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:02.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:02.590+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:02.554+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:02.554+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:02.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:02.554+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:02.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:02.555+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:02.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:02.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:02.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:02.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:02.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:02.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:02.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:02.459+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:03.465 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:03.162+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 11333 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:03.520+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:03.520+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:03.520+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:03.521+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:03.550+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:03.550+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:03.550+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:03.551+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:03.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:03.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:03.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:03.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:03.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:03.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:03.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:03.465+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:04.575+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:04.575+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:04.575+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:04.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:04.576+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:04.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:04.510+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:04.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:04.510+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:04.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:04.510+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:04.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:04.510+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:04.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:04.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:04.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:04.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:04.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:04.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:04.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:04.430+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:05.513 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:05.391+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:05.514 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:05.391+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:05.514 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:05.391+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:05.514 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:05.392+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:05.512+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:05.512+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:05.512+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:05.513+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:05.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:05.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:05.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:05.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:05.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:05.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:05.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:05.543+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:06.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:06.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:06.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:06.476+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:06.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:06.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:06.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:06.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:06.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:06.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:06.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:06.547+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:06.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:06.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:06.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:06.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:06.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:06.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:06.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:06.399+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:07.428+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:07.428+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:07.428+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:07.428+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:07.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:07.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:07.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:07.537+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:07.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:07.537+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:07.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:07.537+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:07.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:07.406+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:07.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:07.406+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:07.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:07.406+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:07.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:07.407+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:08.425 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:08.163+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 11424 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:08.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:08.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:08.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:08.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:08.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:08.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:08.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:08.543+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:08.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:08.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:08.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:08.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:08.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:08.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:08.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:08.424+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:08.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:08.428+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:08.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:08.428+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:08.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:08.428+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:08.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:08.429+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:09.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:09.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:09.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:09.431+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:09.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:09.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:09.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:09.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:09.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:09.589+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:09.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:09.590+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:09.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:09.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:09.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:09.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:09.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:09.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:09.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:09.381+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:10.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:10.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:10.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:10.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:10.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:10.603+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:10.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:10.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:10.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:10.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:10.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:10.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:10.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:10.425+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:10.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:10.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:10.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:10.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:10.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:10.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:10.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:10.420+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:11.610 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:11.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:11.610 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:11.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:11.611 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:11.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:11.611 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:11.453+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:11.611 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:11.386+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:11.611 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:11.386+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:11.611 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:11.386+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:11.611 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:11.387+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:12.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:11.609+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:12.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:11.609+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:12.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:11.609+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:12.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:11.610+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:12.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:12.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:12.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:12.606+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:12.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:12.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:12.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:12.359+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:12.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:12.425+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:12.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:12.425+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:12.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:12.425+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:12.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:12.426+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:13.576 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:13.165+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 11519 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:13.577 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:13.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:13.577 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:13.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:13.577 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:13.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:13.577 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:13.351+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:13.578 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:13.387+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:13.578 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:13.387+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:13.578 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:13.387+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:13.578 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:13.388+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:13.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:13.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:13.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:13.604+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:14.379 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:14.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:14.379 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:14.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:14.379 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:14.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:14.379 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:14.352+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:14.616+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:14.616+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:14.616+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:14.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:14.617+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:14.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:14.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:14.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:14.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:14.378+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:15.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:15.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:15.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:15.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:15.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:15.392+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:15.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:15.393+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:15.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:15.326+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:15.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:15.326+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:15.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:15.326+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:15.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:15.326+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:16.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:15.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:16.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:15.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:16.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:15.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:16.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:15.648+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:16.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:16.426+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:16.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:16.426+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:16.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:16.426+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:16.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:16.427+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:16.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:16.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:16.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:16.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:16.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:16.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:16.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:16.333+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:17.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:16.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:17.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:16.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:17.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:16.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:17.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:16.630+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:17.352+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:17.352+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:17.352+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:17.353+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:17.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:17.585+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:17.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:17.585+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:17.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:17.585+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:17.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:17.586+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:17.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:17.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:17.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:17.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:17.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:17.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:17.412+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:18.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:18.166+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 11611 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:18.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:18.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:18.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:18.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:18.371+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:18.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:18.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:18.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:18.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:18.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:18.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:18.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:18.365+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:18.635+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:18.635+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:18.635+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:18.636+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:19.662 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:19.354+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:19.663 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:19.354+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:19.663 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:19.354+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:19.663 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:19.354+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:19.663 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:19.411+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:19.663 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:19.411+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:19.664 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:19.411+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:19.664 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:19.412+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:19.916 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:19.661+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:19.916 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:19.661+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:19.917 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:19.661+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:19.917 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:19.662+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:20.572 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:20.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:20.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:20.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:20.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:20.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:20.573 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:20.352+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:20.573 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:20.446+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:20.573 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:20.446+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:20.573 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:20.446+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:20.573 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:20.447+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:20.706+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:20.706+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:20.706+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:20.707+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:21.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:21.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:21.341+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:21.342+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:21.439+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:21.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:21.439+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:21.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:21.439+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:21.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:21.439+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:21.692+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:21.692+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:21.692+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:21.693+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:22.302+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:22.302+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:22.302+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:22.303+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:22.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:22.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:22.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:22.465+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:22.662+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:22.662+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:22.662+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:22.663+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:23.515 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:23.168+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 11704 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:23.515 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:23.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:23.516 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:23.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:23.516 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:23.323+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:23.516 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:23.324+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:23.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:23.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:23.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:23.678+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:23.514+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:23.514+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:23.514+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:23.515+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:24.300+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:24.300+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:24.300+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:24.301+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:24.513+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:24.513+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:24.513+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:24.514+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:25.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:24.643+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:25.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:24.643+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:25.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:24.643+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:25.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:24.644+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:25.287+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:25.287+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:25.287+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:25.288+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:25.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:25.480+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:25.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:25.480+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:25.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:25.480+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:25.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:25.481+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:26.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:25.615+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:26.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:25.615+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:26.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:25.615+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:26.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:25.616+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:26.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:26.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:26.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:26.294+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:26.510+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:26.510+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:26.510+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:26.511+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:26.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:26.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:27.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:26.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:27.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:26.641+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:27.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:27.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:27.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:27.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:27.335+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:27.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:27.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:27.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:27.547+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:27.664+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:27.664+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:27.664+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:28.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:27.665+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:28.574 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:28.169+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 11795 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:28.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:28.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:28.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:28.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:28.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:28.371+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:28.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:28.372+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:28.660+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:28.660+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:28.660+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:28.661+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:28.573+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:28.573+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:28.573+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:28.574+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:29.659 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:29.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:29.659 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:29.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:29.659 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:29.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:29.660 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:29.376+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:29.660 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:29.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:29.660 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:29.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:29.660 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:29.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:29.660 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:29.600+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:30.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:29.658+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:30.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:29.658+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:30.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:29.658+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:30.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:29.659+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:30.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:30.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:30.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:30.392+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:30.954 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:30.646+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:30.954 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:30.646+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:30.954 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:30.646+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:30.954 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:30.646+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:30.955 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:30.636+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:30.955 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:30.636+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:30.955 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:30.636+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:30.955 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:30.637+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:31.521 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:31.361+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:31.521 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:31.361+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:31.521 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:31.361+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:31.521 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:31.362+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:31.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:31.663+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:31.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:31.663+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:31.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:31.663+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:31.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:31.664+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:31.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:31.597+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:31.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:31.597+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:31.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:31.597+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:31.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:31.597+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:32.587 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:32.326+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:32.587 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:32.326+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:32.587 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:32.326+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:32.587 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:32.326+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:32.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:32.712+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:32.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:32.712+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:32.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:32.712+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:32.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:32.713+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:32.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:32.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:32.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:32.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:32.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:32.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:32.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:32.619+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:33.578 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:33.170+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 11889 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:33.578 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:33.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:33.578 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:33.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:33.579 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:33.283+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:33.579 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:33.284+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:33.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:33.720+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:33.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:33.720+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:33.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:33.720+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:33.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:33.721+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:33.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:33.579+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:33.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:33.579+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:33.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:33.579+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:33.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:33.580+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:34.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:34.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:34.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:34.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:34.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:34.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:34.597 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:34.329+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:34.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:34.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:34.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:34.596+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:34.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:34.721+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:34.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:34.721+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:34.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:34.721+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:34.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:34.723+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:35.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:35.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:35.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:35.313+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:35.932 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:35.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:35.932 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:35.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:35.932 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:35.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:35.933 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:35.615+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:35.933 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:35.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:35.933 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:35.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:35.933 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:35.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:35.933 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:35.729+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:36.352+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:36.352+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:36.352+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:36.353+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:37.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:36.632+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:37.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:36.632+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:37.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:36.632+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:37.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:36.632+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:37.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:36.738+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:37.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:36.738+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:37.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:36.738+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:37.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:36.740+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:37.764 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:37.602+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:37.764 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:37.602+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:37.765 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:37.602+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:37.765 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:37.603+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:37.765 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:37.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:37.766 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:37.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:37.766 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:37.377+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:37.766 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:37.378+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:38.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:37.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:38.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:37.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:38.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:37.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:38.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:37.763+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:38.561 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:38.171+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 11981 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:38.562 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:38.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:38.562 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:38.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:38.562 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:38.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:38.562 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:38.375+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:38.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:38.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:38.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:38.561+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:38.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:38.795+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:38.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:38.795+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:38.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:38.795+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:38.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:38.796+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:39.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:39.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:39.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:39.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:39.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:39.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:39.788 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:39.531+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:39.788 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:39.390+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:39.789 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:39.390+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:39.789 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:39.390+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:39.789 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:39.390+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:39.784+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:39.784+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:39.784+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:39.785+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:40.741 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:40.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:40.742 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:40.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:40.742 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:40.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:40.742 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:40.425+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:40.742 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:40.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:40.742 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:40.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:40.742 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:40.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:40.742 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:40.525+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:41.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:40.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:41.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:40.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:41.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:40.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:41.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:40.740+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:41.737 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:41.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:41.737 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:41.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:41.737 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:41.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:41.737 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:41.492+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:41.737 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:41.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:41.738 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:41.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:41.738 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:41.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:41.738 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:41.455+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:42.076 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:41.734+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:42.076 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:41.734+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:42.076 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:41.734+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:42.076 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:41.735+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:42.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:42.690+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:42.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:42.690+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:42.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:42.690+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:42.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:42.690+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:42.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:42.455+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:42.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:42.455+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:42.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:42.455+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:42.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:42.455+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:42.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:42.478+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:42.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:42.478+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:42.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:42.478+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:42.806 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:42.478+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:43.432 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:43.173+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 12074 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:43.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:43.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:43.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:43.649+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:43.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:43.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:43.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:43.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:43.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:43.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:43.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:43.432+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:43.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:43.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:43.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:43.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:43.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:43.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:43.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:43.528+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:44.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:44.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:44.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:44.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:44.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:44.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:44.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:44.484+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:44.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:44.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:44.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:44.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:44.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:44.627+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:44.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:44.628+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:44.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:44.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:44.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:44.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:44.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:44.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:44.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:44.405+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:45.460 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:45.394+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:45.460 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:45.394+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:45.461 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:45.394+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:45.461 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:45.395+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:45.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:45.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:45.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:45.460+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:45.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:45.585+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:45.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:45.585+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:45.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:45.585+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:45.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:45.586+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:46.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:46.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:46.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:46.373+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:46.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:46.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:46.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:46.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:46.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:46.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:46.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:46.412+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:46.540+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:46.540+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:46.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:46.540+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:46.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:46.541+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:47.389+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:47.389+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:47.389+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:47.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:47.390+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:47.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:47.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:47.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:47.388+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:47.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:47.492+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:47.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:47.492+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:47.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:47.492+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:47.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:47.493+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:48.445 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:48.174+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 12166 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:48.446 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:48.408+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:48.446 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:48.408+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:48.446 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:48.408+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:48.446 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:48.409+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:48.446 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:48.367+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:48.447 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:48.367+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:48.447 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:48.367+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:48.447 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:48.368+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:48.444+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:48.444+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:48.444+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:48.445+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:49.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:49.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:49.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:49.395+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:49.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:49.435+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:49.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:49.435+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:49.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:49.435+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:49.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:49.436+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:49.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:49.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:49.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:49.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:49.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:49.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:49.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:49.459+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:50.423+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:50.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:50.423+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:50.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:50.423+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:50.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:50.424+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:50.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:50.406+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:50.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:50.406+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:50.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:50.406+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:50.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:50.407+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:50.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:50.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:50.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:50.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:50.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:50.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:50.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:50.506+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:51.429+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:51.429+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:51.429+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:51.430+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:51.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:51.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:51.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:51.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:51.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:51.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:51.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:51.380+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:51.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:51.512+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:51.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:51.512+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:51.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:51.512+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:51.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:51.513+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:52.413+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:52.413+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:52.413+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:52.414+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:52.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:52.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:52.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:52.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:52.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:52.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:52.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:52.476+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:52.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:52.533+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:52.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:52.533+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:52.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:52.533+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:52.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:52.534+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:53.502 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:53.176+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 12260 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:53.502 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:53.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:53.502 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:53.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:53.503 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:53.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:53.503 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:53.408+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:53.501+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:53.501+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:53.501+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:53.502+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:53.541+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:53.541+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:53.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:53.541+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:53.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:53.542+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:54.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:54.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:54.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:54.428+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:54.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:54.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:54.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:54.568+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:54.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:54.487+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:54.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:54.487+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:54.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:54.487+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:54.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:54.488+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:55.428+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:55.428+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:55.428+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:55.429+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:55.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:55.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:55.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:55.449+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:55.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:55.543+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:55.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:55.543+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:55.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:55.543+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:55.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:55.544+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:56.414+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:56.414+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:56.414+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:56.415+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:56.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:56.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:56.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:56.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:56.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:56.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:56.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:56.382+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:56.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:56.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:56.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:56.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:56.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:56.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:56.500+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:57.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:57.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:57.499+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:57.500+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:57.360+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:57.360+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:57.360+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:57.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:57.361+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:57.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:57.375+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:57.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:57.375+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:57.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:57.375+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:57.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:57.376+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:58.465 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:31:58.177+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 12353 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:31:58.465 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:58.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:58.466 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:58.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:58.466 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:58.379+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:58.466 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:58.380+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:58.466 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:58.315+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:58.466 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:58.315+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:58.466 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:58.315+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:58.467 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:58.316+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:58.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:58.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:58.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:58.465+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:59.320+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:59.320+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:59.320+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:31:59.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:31:59.321+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:59.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:59.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:59.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:31:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:31:59.422+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:31:59.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:59.418+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:59.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:59.418+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:59.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:59.418+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:31:59.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:31:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:31:59.419+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:00.435 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:00.278+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:00.435 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:00.278+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:00.436 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:00.278+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:00.436 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:00.279+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:00.434+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:00.434+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:00.434+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:00.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:00.435+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:00.451+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:00.451+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:00.451+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:00.451+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:01.474+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:01.474+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:01.474+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:01.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:01.475+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:01.245+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:01.245+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:01.245+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:01.246+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:01.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:01.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:01.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:01.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:01.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:01.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:01.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:01.457+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:02.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:02.252+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:02.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:02.252+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:02.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:02.252+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:02.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:02.253+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:02.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:02.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:02.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:02.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:02.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:02.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:02.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:02.452+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:02.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:02.500+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:02.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:02.500+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:02.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:02.500+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:02.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:02.501+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:03.428 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:03.178+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 12446 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:03.428 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:03.229+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:03.429 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:03.229+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:03.429 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:03.229+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:03.429 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:03.230+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:03.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:03.510+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:03.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:03.510+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:03.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:03.510+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:03.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:03.510+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:03.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:03.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:03.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:03.427+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:04.232+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:04.233+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:04.233+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:04.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:04.233+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:04.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:04.443+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:04.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:04.443+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:04.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:04.443+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:04.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:04.444+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:04.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:04.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:04.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:04.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:04.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:04.464+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:04.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:04.465+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:05.466 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:05.196+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:05.467 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:05.196+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:05.467 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:05.196+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:05.467 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:05.197+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:05.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:05.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:05.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:05.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:05.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:05.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:05.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:05.476+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:05.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:05.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:05.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:05.466+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:06.430 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:06.156+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:06.431 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:06.156+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:06.431 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:06.156+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:06.431 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:06.157+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:06.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:06.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:06.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:06.430+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:06.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:06.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:06.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:06.457+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:07.412 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:07.143+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:07.413 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:07.143+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:07.413 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:07.143+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:07.413 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:07.143+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:07.413 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:07.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:07.413 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:07.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:07.414 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:07.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:07.414 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:07.384+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:07.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:07.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:07.411+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:07.412+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:08.366 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:08.180+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 12538 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:08.366 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:08.338+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:08.366 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:08.338+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:08.366 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:08.338+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:08.367 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:08.339+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:08.367 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:08.107+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:08.367 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:08.107+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:08.367 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:08.107+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:08.367 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:08.107+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:08.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:08.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:08.365+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:08.366+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:09.403 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:09.106+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:09.403 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:09.106+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:09.404 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:09.106+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:09.404 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:09.107+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:09.404 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:09.353+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:09.404 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:09.353+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:09.404 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:09.353+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:09.404 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:09.354+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:09.402+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:09.402+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:09.402+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:09.403+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:10.429 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:10.311+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:10.430 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:10.311+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:10.430 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:10.311+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:10.430 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:10.312+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:10.430 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:10.143+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:10.430 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:10.143+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:10.430 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:10.143+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:10.431 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:10.144+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:10.428+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:10.428+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:10.428+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:10.429+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:11.463 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:11.172+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:11.463 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:11.172+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:11.463 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:11.172+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:11.463 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:11.172+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:11.464 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:11.273+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:11.464 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:11.273+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:11.464 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:11.273+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:11.464 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:11.274+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:11.461+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:11.461+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:11.461+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:11.462+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:12.453 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:12.144+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:12.454 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:12.144+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:12.454 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:12.144+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:12.454 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:12.145+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:12.454 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:12.313+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:12.454 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:12.313+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:12.454 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:12.313+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:12.455 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:12.314+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:12.796 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:12.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:12.796 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:12.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:12.796 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:12.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:12.796 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:12.453+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:13.473 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:13.181+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 12632 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:13.473 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:13.176+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:13.474 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:13.176+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:13.474 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:13.176+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:13.474 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:13.177+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:13.474 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:13.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:13.474 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:13.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:13.474 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:13.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:13.475 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:13.345+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:13.472+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:13.472+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:13.472+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:13.473+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:14.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:14.373+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:14.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:14.373+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:14.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:14.373+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:14.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:14.373+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:14.201+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:14.201+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:14.201+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:14.202+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:14.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:14.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:14.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:14.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:14.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:14.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:14.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:14.425+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:15.180+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:15.180+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:15.180+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:15.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:15.181+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:15.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:15.422+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:15.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:15.422+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:15.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:15.422+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:15.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:15.423+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:15.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:15.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:15.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:15.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:15.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:15.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:15.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:15.367+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:16.215+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:16.215+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:16.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:16.215+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:16.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:16.216+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:16.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:16.363+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:16.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:16.363+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:16.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:16.363+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:16.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:16.363+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:16.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:16.450+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:16.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:16.450+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:16.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:16.450+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:16.609 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:16.451+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:17.325+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:17.325+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:17.325+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:17.326+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:17.440+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:17.440+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:17.440+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:17.441+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:17.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:17.262+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:17.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:17.262+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:17.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:17.262+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:17.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:17.263+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:18.438 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:18.182+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 12725 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:18.439 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:18.236+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:18.439 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:18.236+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:18.439 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:18.236+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:18.439 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:18.237+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:18.439 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:18.305+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:18.439 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:18.305+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:18.440 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:18.305+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:18.440 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:18.306+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:18.437+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:18.437+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:18.437+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:18.438+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:19.250+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:19.250+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:19.250+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:19.251+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:19.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:19.436+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:19.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:19.436+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:19.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:19.436+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:19.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:19.436+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:19.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:19.284+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:19.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:19.284+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:19.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:19.284+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:19.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:19.284+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:20.486 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:20.248+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:20.487 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:20.248+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:20.487 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:20.248+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:20.487 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:20.249+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:20.487 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:20.331+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:20.487 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:20.331+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:20.488 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:20.331+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:20.488 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:20.332+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:20.485+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:20.485+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:20.485+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:20.486+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:21.265+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:21.265+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:21.265+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:21.266+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:21.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:21.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:21.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:21.440+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:21.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:21.320+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:21.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:21.320+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:21.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:21.320+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:21.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:21.320+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:22.331 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:22.235+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:22.331 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:22.235+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:22.332 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:22.235+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:22.332 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:22.236+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:22.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:22.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:22.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:22.455+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:22.330+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:22.330+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:22.330+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:22.331+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:23.500 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:23.184+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 12818 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:23.500 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:23.193+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:23.501 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:23.193+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:23.501 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:23.193+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:23.501 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:23.194+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:23.501 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:23.334+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:23.501 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:23.334+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:23.501 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:23.334+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:23.501 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:23.334+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:23.498+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:23.498+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:23.498+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:23.499+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:24.533 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:24.185+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:24.533 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:24.185+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:24.533 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:24.185+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:24.533 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:24.186+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:24.533 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:24.302+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:24.534 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:24.302+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:24.534 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:24.302+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:24.534 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:24.303+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:24.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:24.530+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:24.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:24.530+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:24.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:24.530+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:24.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:24.531+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:25.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:25.168+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:25.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:25.168+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:25.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:25.168+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:25.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:25.169+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:25.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:25.273+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:25.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:25.273+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:25.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:25.273+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:25.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:25.273+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:25.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:25.548+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:25.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:25.548+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:25.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:25.548+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:25.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:25.549+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:26.547 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:26.149+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:26.548 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:26.149+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:26.548 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:26.149+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:26.548 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:26.149+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:26.548 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:26.236+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:26.548 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:26.236+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:26.548 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:26.236+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:26.549 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:26.237+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:26.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:26.545+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:26.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:26.545+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:26.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:26.545+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:26.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:26.546+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:27.502 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:27.103+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:27.502 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:27.103+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:27.502 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:27.104+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:27.502 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:27.104+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:27.503 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:27.260+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:27.503 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:27.260+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:27.503 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:27.260+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:27.503 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:27.260+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:27.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:27.500+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:27.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:27.500+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:27.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:27.500+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:27.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:27.501+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:28.491 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:28.185+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 12910 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:28.492 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:28.260+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:28.492 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:28.260+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:28.492 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:28.260+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:28.492 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:28.262+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:28.492 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:28.138+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:28.492 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:28.138+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:28.493 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:28.138+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:28.493 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:28.138+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:28.490+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:28.490+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:28.490+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:28.491+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:29.253+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:29.253+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:29.253+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:29.254+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:29.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:29.097+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:29.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:29.097+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:29.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:29.097+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:29.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:29.097+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:29.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:29.519+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:29.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:29.519+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:29.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:29.519+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:29.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:29.520+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:30.491 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:30.119+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:30.491 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:30.119+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:30.492 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:30.119+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:30.492 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:30.119+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:30.492 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:30.249+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:30.492 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:30.249+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:30.492 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:30.249+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:30.492 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:30.250+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:30.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:30.490+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:30.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:30.490+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:30.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:30.490+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:30.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:30.491+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:31.446 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:31.115+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:31.446 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:31.115+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:31.446 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:31.115+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:31.446 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:31.115+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:31.447 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:31.276+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:31.447 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:31.276+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:31.447 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:31.276+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:31.447 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:31.277+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:31.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:31.445+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:31.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:31.445+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:31.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:31.445+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:31.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:31.446+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:32.302+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:32.302+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:32.302+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:32.302+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:32.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:32.087+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:32.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:32.087+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:32.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:32.087+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:32.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:32.087+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:32.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:32.453+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:32.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:32.453+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:32.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:32.453+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:32.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:32.454+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:33.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:33.187+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 13004 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:33.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:33.077+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:33.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:33.077+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:33.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:33.077+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:33.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:33.078+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:33.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:33.312+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:33.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:33.312+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:33.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:33.312+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:33.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:33.313+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:33.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:33.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:33.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:33.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:33.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:33.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:33.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:33.455+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:34.300 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:34.035+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:34.300 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:34.035+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:34.301 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:34.035+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:34.301 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:34.036+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:34.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:34.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:34.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:34.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:34.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:34.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:34.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:34.333+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:34.426+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:34.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:34.426+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:34.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:34.426+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:34.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:34.427+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:35.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:35.018+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:35.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:35.018+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:35.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:35.018+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:35.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:35.018+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:35.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:35.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:35.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:35.355+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:35.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:35.473+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:35.473+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:35.473+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:35.474+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:36.334 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:36.016+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:36.334 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:36.016+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:36.334 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:36.016+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:36.335 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:36.016+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:36.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:36.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:36.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:36.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:36.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:36.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:36.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:36.334+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:36.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:36.441+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:36.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:36.441+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:36.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:36.441+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:36.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:36.442+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:37.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:37.000+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:37.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:37.000+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:37.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:37.000+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:37.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:37.000+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:37.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:37.482+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:37.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:37.482+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:37.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:37.482+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:37.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:37.482+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:37.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:37.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:37.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:37.359+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:38.313 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:38.188+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 13097 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:38.313 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:38.003+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:38.314 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:38.003+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:38.314 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:38.003+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:38.314 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:38.003+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:38.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:38.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:38.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:38.354+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:38.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:38.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:38.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:38.449+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:39.347 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:39.044+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:39.348 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:39.044+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:39.348 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:39.044+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:39.348 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:39.044+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:39.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:39.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:39.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:39.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:39.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:39.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:39.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:39.347+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:39.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:39.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:39.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:39.466+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:39.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:39.466+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:39.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:39.466+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:40.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:40.045+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:40.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:40.045+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:40.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:40.045+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:40.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:40.046+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:40.500+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:40.500+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:40.500+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:40.500+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:40.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:40.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:40.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:40.358+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:41.348 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:41.067+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:41.348 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:41.067+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:41.348 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:41.067+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:41.348 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:41.067+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:41.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:41.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:41.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:41.347+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:41.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:41.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:41.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:41.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:41.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:41.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:41.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:41.534+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:42.383 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:42.116+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:42.383 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:42.116+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:42.383 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:42.116+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:42.383 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:42.116+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:42.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:42.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:42.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:42.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:42.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:42.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:42.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:42.509+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:42.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:42.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:42.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:42.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:42.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:42.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:42.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:42.382+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:43.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:43.189+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 13190 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:43.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:43.098+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:43.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:43.098+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:43.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:43.098+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:43.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:43.098+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:43.385+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:43.385+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:43.385+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:43.385+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:43.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:43.506+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:43.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:43.506+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:43.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:43.506+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:43.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:43.506+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:44.350 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:44.080+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:44.350 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:44.080+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:44.350 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:44.080+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:44.350 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:44.081+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:44.457+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:44.457+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:44.457+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:44.458+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:44.348+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:44.348+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:44.348+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:44.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:44.349+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:45.064+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:45.064+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:45.064+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:45.065+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:45.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:45.377+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:45.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:45.377+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:45.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:45.377+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:45.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:45.378+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:45.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:45.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:45.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:45.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:45.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:45.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:45.484+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:46.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:46.080+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:46.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:46.080+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:46.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:46.080+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:46.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:46.081+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:46.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:46.476+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:46.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:46.476+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:46.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:46.476+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:46.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:46.476+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:46.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:46.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:46.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:46.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:46.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:46.384+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:47.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:47.084+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:47.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:47.084+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:47.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:47.084+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:47.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:47.085+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:47.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:47.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:47.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:47.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:47.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:47.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:47.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:47.484+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:47.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:47.424+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:47.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:47.424+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:47.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:47.424+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:47.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:47.425+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:48.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:48.191+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 13282 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:48.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:48.069+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:48.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:48.069+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:48.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:48.069+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:48.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:48.070+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:48.488+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:48.488+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:48.488+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:48.489+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:48.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:48.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:48.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:48.400+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:49.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:49.055+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:49.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:49.055+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:49.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:49.055+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:49.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:49.056+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:49.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:49.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:49.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:49.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:49.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:49.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:49.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:49.527+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:49.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:49.426+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:49.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:49.426+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:49.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:49.426+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:49.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:49.427+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:50.380 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:50.105+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:50.381 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:50.105+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:50.381 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:50.105+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:50.381 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:50.106+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:50.526+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:50.526+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:50.526+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:50.527+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:50.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:50.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:50.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:50.380+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:51.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:51.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:51.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:51.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:51.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:51.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:51.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:51.357+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:51.574 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:51.108+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:51.575 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:51.108+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:51.575 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:51.108+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:51.575 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:51.109+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:51.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:51.572+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:51.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:51.572+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:51.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:51.572+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:51.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:51.573+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:52.580 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:52.116+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:52.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:52.116+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:52.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:52.116+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:52.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:52.116+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:52.581 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:52.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:52.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:52.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:52.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:52.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:52.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:52.329+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:52.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:52.579+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:52.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:52.579+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:52.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:52.579+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:52.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:52.580+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:53.565 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:53.192+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 13376 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:53.566 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:53.161+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:53.566 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:53.161+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:53.566 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:53.161+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:53.566 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:53.162+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:53.566 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:53.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:53.567 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:53.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:53.567 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:53.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:53.567 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:53.365+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:53.564+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:53.564+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:53.564+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:53.565+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:54.534 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:54.176+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:54.534 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:54.176+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:54.534 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:54.176+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:54.534 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:54.177+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:54.535 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:54.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:54.535 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:54.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:54.535 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:54.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:54.535 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:54.405+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:54.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:54.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:54.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:54.533+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:55.524 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:55.403+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:55.524 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:55.403+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:55.524 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:55.403+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:55.524 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:55.403+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:55.525 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:55.169+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:55.525 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:55.169+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:55.525 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:55.169+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:55.525 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:55.170+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:55.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:55.523+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:55.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:55.523+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:55.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:55.523+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:55.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:55.523+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:56.555 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:56.197+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:56.556 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:56.197+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:56.556 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:56.197+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:56.556 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:56.198+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:56.556 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:56.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:56.557 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:56.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:56.557 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:56.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:56.557 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:56.408+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:56.554+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:56.554+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:56.554+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:56.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:56.555+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:57.544 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:57.209+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:57.544 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:57.209+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:57.544 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:57.209+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:57.544 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:57.210+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:57.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:57.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:57.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:57.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:57.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:57.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:57.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:57.435+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:57.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:57.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:57.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:57.543+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:58.588 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:32:58.194+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 13469 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:32:58.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:58.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:58.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:58.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:58.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:58.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:58.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:58.438+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:58.590 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:58.205+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:58.590 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:58.205+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:58.590 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:58.205+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:58.590 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:58.206+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:58.587+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:58.587+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:58.587+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:58.588+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:59.476 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:59.224+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:59.476 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:59.224+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:59.476 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:59.224+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:32:59.476 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:32:59.225+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:59.570+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:59.570+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:59.570+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:32:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:32:59.571+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:32:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:59.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:59.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:59.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:32:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:32:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:32:59.475+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:00.494 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:00.231+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:00.494 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:00.231+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:00.494 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:00.231+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:00.494 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:00.232+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:00.614+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:00.614+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:00.614+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:00.615+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:00.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:00.493+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:00.493+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:00.493+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:00.494+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:01.540 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:01.189+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:01.540 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:01.189+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:01.540 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:01.189+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:01.540 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:01.190+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:01.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:01.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:01.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:01.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:01.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:01.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:01.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:01.585+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:01.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:01.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:01.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:01.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:01.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:01.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:01.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:01.539+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:02.559 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:02.204+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:02.559 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:02.204+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:02.559 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:02.204+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:02.560 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:02.204+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:02.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:02.576+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:02.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:02.576+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:02.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:02.576+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:02.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:02.577+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:02.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:02.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:02.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:02.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:02.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:02.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:02.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:02.559+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:03.541 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:03.195+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 13562 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:03.541 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:03.240+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:03.541 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:03.240+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:03.542 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:03.240+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:03.542 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:03.240+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:03.542 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:03.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:03.542 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:03.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:03.542 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:03.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:03.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:03.540+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:03.567+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:03.567+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:03.567+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:03.568+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:04.524 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:04.230+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:04.524 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:04.230+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:04.524 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:04.230+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:04.524 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:04.231+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:04.577+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:04.577+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:04.577+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:04.578+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:04.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:04.522+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:04.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:04.522+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:04.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:04.522+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:04.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:04.523+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:05.192+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:05.192+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:05.192+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:05.193+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:05.557+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:05.557+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:05.557+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:05.558+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:05.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:05.562+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:05.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:05.562+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:05.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:05.562+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:05.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:05.563+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:06.548 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:06.159+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:06.549 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:06.159+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:06.549 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:06.159+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:06.549 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:06.159+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:06.547+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:06.547+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:06.547+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:06.548+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:06.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:06.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:06.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:06.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:06.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:06.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:06.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:06.561+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:07.529 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:07.188+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:07.529 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:07.188+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:07.529 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:07.188+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:07.530 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:07.189+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:07.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:07.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:07.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:07.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:07.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:07.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:07.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:07.591+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:07.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:07.528+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:07.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:07.528+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:07.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:07.528+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:07.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:07.529+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:08.519 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:08.197+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 13654 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:08.520 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:08.185+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:08.520 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:08.185+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:08.520 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:08.185+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:08.520 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:08.185+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:08.545+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:08.545+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:08.545+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:08.546+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:08.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:08.518+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:08.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:08.518+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:08.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:08.518+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:08.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:08.519+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:09.503 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:09.217+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:09.503 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:09.217+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:09.503 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:09.217+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:09.503 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:09.217+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:09.513+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:09.513+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:09.513+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:09.514+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:09.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:09.500+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:09.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:09.501+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:09.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:09.501+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:09.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:09.502+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:10.470 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:10.245+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:10.470 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:10.245+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:10.470 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:10.245+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:10.471 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:10.246+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:10.467+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:10.467+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:10.467+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:10.468+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:10.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:10.543+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:10.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:10.543+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:10.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:10.543+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:10.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:10.543+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:11.496 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:11.256+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:11.497 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:11.256+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:11.497 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:11.256+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:11.497 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:11.257+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:11.494+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:11.494+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:11.494+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:11.495+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:11.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:11.550+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:11.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:11.550+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:11.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:11.550+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:11.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:11.550+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:12.575 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:12.287+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:12.576 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:12.287+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:12.576 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:12.287+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:12.576 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:12.288+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:12.576 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:12.463+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:12.577 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:12.463+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:12.577 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:12.463+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:12.577 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:12.464+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:12.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:12.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:12.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:12.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:12.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:12.574+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:12.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:12.574+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:13.482 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:13.197+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 13748 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:13.482 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:13.319+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:13.483 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:13.319+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:13.483 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:13.319+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:13.483 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:13.320+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:13.481+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:13.481+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:13.481+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:13.482+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:13.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:13.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:13.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:13.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:13.567+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:14.496 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:14.355+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:14.496 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:14.355+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:14.496 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:14.355+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:14.496 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:14.355+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:14.495+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:14.495+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:14.495+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:14.495+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:14.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:14.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:14.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:14.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:14.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:14.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:14.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:14.536+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:15.389 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:15.387+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:15.390 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:15.387+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:15.390 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:15.387+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:15.390 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:15.388+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:15.535+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:15.535+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:15.535+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:15.536+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:15.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:15.577+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:15.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:15.577+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:15.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:15.577+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:15.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:15.578+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:16.350+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:16.350+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:16.350+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:16.351+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:16.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:16.490+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:16.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:16.490+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:16.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:16.490+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:16.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:16.491+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:16.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:16.591+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:16.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:16.591+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:16.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:16.591+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:16.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:16.592+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:17.378+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:17.378+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:17.378+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:17.379+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:17.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:17.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:17.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:17.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:17.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:17.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:17.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:17.466+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:17.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:17.604+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:17.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:17.604+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:17.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:17.604+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:17.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:17.604+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:18.507 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:18.200+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 13843 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:18.507 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:18.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:18.507 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:18.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:18.507 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:18.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:18.508 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:18.371+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:18.506+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:18.506+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:18.506+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:18.506+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:18.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:18.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:18.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:18.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:18.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:18.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:18.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:18.611+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:19.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:19.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:19.388+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:19.389+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:19.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:19.603+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:19.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:19.603+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:19.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:19.603+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:19.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:19.604+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:19.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:19.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:19.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:19.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:19.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:19.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:19.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:19.524+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:20.567 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:20.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:20.567 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:20.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:20.567 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:20.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:20.567 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:20.382+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:20.608+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:20.608+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:20.608+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:20.609+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:20.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:20.566+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:20.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:20.566+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:20.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:20.566+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:20.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:20.567+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:21.687 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:21.371+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:21.688 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:21.371+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:21.688 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:21.371+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:21.688 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:21.371+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:21.688 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:21.556+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:21.688 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:21.556+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:21.689 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:21.556+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:21.689 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:21.557+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:21.689 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:21.587+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:21.689 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:21.587+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:21.689 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:21.587+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:21.689 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:21.588+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:22.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:22.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:22.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:22.528+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:22.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:22.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:22.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:22.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:22.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:22.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:22.326+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:22.604+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:22.604+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:22.604+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:22.605+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:23.579 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:23.201+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 13932 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:23.579 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:23.319+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:23.579 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:23.319+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:23.580 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:23.319+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:23.580 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:23.320+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:23.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:23.579+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:23.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:23.579+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:23.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:23.579+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:23.580+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:23.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:23.578+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:23.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:23.578+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:23.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:23.578+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:23.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:23.578+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:24.567+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:24.567+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:24.567+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:24.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:24.567+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:24.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:24.350+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:24.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:24.350+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:24.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:24.350+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:24.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:24.351+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:24.584+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:24.584+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:24.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:24.584+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:24.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:24.585+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:25.594 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:25.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:25.595 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:25.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:25.595 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:25.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:25.595 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:25.344+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:25.595 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:25.535+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:25.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:25.535+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:25.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:25.535+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:25.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:25.536+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:25.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:25.593+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:25.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:25.593+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:25.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:25.593+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:25.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:25.594+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:26.631 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:26.363+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:26.631 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:26.363+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:26.631 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:26.363+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:26.631 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:26.364+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:26.632 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:26.532+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:26.632 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:26.532+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:26.632 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:26.532+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:26.632 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:26.533+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:26.630+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:26.630+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:26.630+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:26.631+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:27.663 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:27.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:27.663 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:27.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:27.663 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:27.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:27.663 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:27.528+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:27.664 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:27.392+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:27.664 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:27.392+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:27.664 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:27.392+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:27.664 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:27.393+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:28.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:27.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:28.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:27.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:28.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:27.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:28.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:27.662+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:28.539 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:28.203+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14019 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:28.540 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:28.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:28.540 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:28.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:28.540 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:28.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:28.540 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:28.419+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:28.538+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:28.538+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:28.538+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:28.539+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:28.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:28.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:28.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:28.669+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:29.692 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:29.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:29.693 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:29.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:29.693 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:29.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:29.693 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:29.418+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:29.693 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:29.504+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:29.693 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:29.504+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:29.694 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:29.504+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:29.694 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:29.504+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:29.691+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:29.691+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:29.691+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:29.692+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:30.681 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:30.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:30.682 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:30.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:30.682 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:30.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:30.682 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:30.416+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:30.682 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:30.546+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:30.682 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:30.546+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:30.682 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:30.546+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:30.683 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:30.547+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:30.680+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:30.680+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:30.680+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:31.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:30.681+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:31.688 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:31.435+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:31.688 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:31.435+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:31.689 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:31.435+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:31.689 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:31.436+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:31.689 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:31.542+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:31.690 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:31.542+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:31.690 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:31.542+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:31.690 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:31.543+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:31.702+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:31.702+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:31.702+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:31.702+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:32.668 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:32.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:32.668 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:32.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:32.668 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:32.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:32.669 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:32.405+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:32.669 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:32.526+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:32.669 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:32.526+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:32.669 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:32.526+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:32.669 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:32.526+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:33.108 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:32.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:33.108 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:32.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:33.108 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:32.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:33.109 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:32.704+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:33.408 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:33.204+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14108 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:33.663 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:33.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:33.663 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:33.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:33.664 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:33.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:33.664 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:33.408+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:33.664 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:33.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:33.664 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:33.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:33.665 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:33.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:33.665 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:33.534+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:33.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:33.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:33.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:33.663+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:34.686 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:34.391+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:34.686 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:34.391+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:34.686 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:34.391+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:34.687 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:34.392+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:34.687 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:34.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:34.687 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:34.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:34.687 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:34.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:34.687 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:34.525+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:34.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:34.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:34.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:34.685+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:35.687 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:35.503+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:35.687 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:35.503+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:35.687 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:35.503+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:35.688 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:35.504+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:35.688 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:35.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:35.688 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:35.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:35.688 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:35.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:35.688 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:35.416+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:36.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:35.686+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:36.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:35.686+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:36.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:35.686+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:36.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:35.687+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:36.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:36.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:36.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:36.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:36.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:36.454+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:36.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:36.455+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:36.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:36.637+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:36.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:36.637+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:36.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:36.637+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:36.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:36.638+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:36.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:36.429+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:36.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:36.429+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:36.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:36.429+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:36.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:36.430+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:37.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:37.499+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:37.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:37.499+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:37.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:37.499+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:37.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:37.500+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:37.595+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:37.595+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:37.595+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:37.595+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:37.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:37.390+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:37.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:37.390+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:37.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:37.390+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:37.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:37.390+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:38.555 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:38.206+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14195 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:38.555 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:38.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:38.555 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:38.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:38.555 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:38.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:38.556 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:38.353+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:38.556 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:38.451+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:38.556 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:38.451+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:38.556 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:38.451+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:38.556 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:38.452+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:38.557 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:38.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:38.557 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:38.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:38.557 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:38.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:38.554+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:39.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:39.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:39.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:39.358+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:39.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:39.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:39.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:39.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:39.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:39.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:39.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:39.431+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:39.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:39.602+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:39.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:39.602+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:39.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:39.602+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:39.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:39.603+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:40.432 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:40.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:40.433 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:40.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:40.433 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:40.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:40.433 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:40.327+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:40.431+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:40.431+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:40.431+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:40.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:40.432+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:40.589+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:40.589+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:40.589+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:40.589+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:41.600 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:41.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:41.601 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:41.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:41.601 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:41.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:41.601 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:41.330+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:41.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:41.476+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:41.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:41.476+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:41.602 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:41.476+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:41.602 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:41.477+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:41.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:41.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:41.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:41.600+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:42.640 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:42.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:42.640 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:42.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:42.640 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:42.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:42.640 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:42.360+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:42.640 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:42.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:42.641 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:42.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:42.641 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:42.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:42.641 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:42.433+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:42.638+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:42.638+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:42.638+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:42.639+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:43.585 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:43.207+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14283 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:43.586 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:43.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:43.586 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:43.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:43.586 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:43.311+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:43.586 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:43.312+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:43.586 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:43.422+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:43.586 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:43.422+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:43.587 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:43.422+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:43.587 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:43.423+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:43.652+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:43.652+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:43.652+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:43.653+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:44.347+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:44.347+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:44.347+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:44.348+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:44.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:44.444+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:44.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:44.444+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:44.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:44.444+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:44.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:44.445+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:44.956 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:44.624+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:44.956 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:44.624+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:44.956 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:44.624+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:44.957 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:44.625+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:45.486+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:45.486+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:45.486+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:45.487+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:45.310+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:45.310+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:45.310+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:45.312+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:45.635+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:45.635+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:45.635+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:45.636+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:46.599 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:46.278+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:46.599 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:46.278+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:46.599 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:46.278+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:46.599 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:46.279+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:46.600 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:46.482+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:46.600 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:46.482+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:46.600 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:46.482+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:46.600 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:46.483+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:46.597+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:46.597+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:46.597+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:46.598+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:47.325+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:47.325+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:47.325+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:47.326+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:47.513+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:47.513+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:47.513+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:47.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:47.514+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:47.604+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:47.604+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:47.605+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:47.605+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:48.520 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:48.209+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14370 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:48.521 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:48.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:48.521 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:48.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:48.521 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:48.362+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:48.522 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:48.363+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:48.615+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:48.615+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:48.615+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:48.616+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:48.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:48.519+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:48.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:48.519+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:48.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:48.519+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:48.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:48.520+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:49.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:49.591+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:49.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:49.591+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:49.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:49.591+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:49.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:49.592+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:49.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:49.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:49.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:49.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:49.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:49.372+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:49.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:49.373+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:49.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:49.477+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:49.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:49.477+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:49.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:49.477+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:49.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:49.478+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:50.517 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:50.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:50.518 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:50.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:50.518 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:50.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:50.518 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:50.359+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:50.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:50.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:50.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:50.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:50.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:50.553+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:50.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:50.516+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:50.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:50.516+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:50.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:50.516+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:50.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:50.517+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:51.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:51.328+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:51.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:51.328+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:51.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:51.328+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:51.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:51.329+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:51.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:51.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:51.546+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:51.546+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:51.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:51.538+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:51.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:51.538+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:51.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:51.538+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:51.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:51.539+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:52.568 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:52.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:52.569 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:52.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:52.569 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:52.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:52.569 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:52.309+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:52.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:52.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:52.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:52.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:52.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:52.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:52.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:52.568+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:52.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:52.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:52.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:52.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:52.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:52.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:52.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:52.582+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:53.559 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:53.210+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14459 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:53.559 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:53.298+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:53.560 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:53.298+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:53.560 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:53.298+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:53.560 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:53.299+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:53.626+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:53.626+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:53.626+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:53.627+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:53.557+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:53.557+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:53.557+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:53.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:53.559+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:54.592 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:54.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:54.592 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:54.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:54.592 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:54.326+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:54.592 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:54.327+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:54.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:54.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:54.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:54.644+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:54.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:54.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:54.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:54.590+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:55.636 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:55.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:55.636 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:55.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:55.636 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:55.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:55.636 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:55.354+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:55.637 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:55.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:55.637 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:55.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:55.637 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:55.599+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:55.637 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:55.600+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:55.633+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:55.633+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:55.633+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:55.635+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:56.336+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:56.336+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:56.336+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:56.337+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:56.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:56.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:56.554+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:56.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:56.554+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:56.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:56.554+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:57.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:56.605+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:57.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:56.605+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:57.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:56.605+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:57.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:56.606+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:57.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:57.313+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:57.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:57.313+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:57.589 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:57.313+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:57.589 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:57.314+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:57.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:57.511+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:57.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:57.511+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:57.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:57.511+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:57.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:57.512+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:57.586+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:57.586+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:57.586+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:57.587+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:58.524 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:33:58.211+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14546 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:33:58.525 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:58.307+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:58.525 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:58.307+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:58.525 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:58.307+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:58.525 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:58.308+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:58.588+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:58.588+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:58.588+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:58.589+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:58.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:58.522+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:58.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:58.522+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:58.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:58.522+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:58.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:58.523+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:59.537 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:59.285+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:59.537 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:59.285+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:59.537 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:59.285+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:33:59.537 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:33:59.286+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:59.584+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:59.584+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:59.584+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:33:59.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:33:59.584+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:33:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:59.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:59.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:59.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:33:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:33:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:33:59.536+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:00.491 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:00.314+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:00.491 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:00.314+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:00.491 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:00.314+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:00.492 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:00.315+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:00.566+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:00.566+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:00.566+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:00.567+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:00.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:00.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:00.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:00.491+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:01.342+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:01.342+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:01.342+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:01.343+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:01.459+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:01.459+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:01.459+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:01.460+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:02.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:01.613+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:02.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:01.613+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:02.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:01.613+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:02.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:01.614+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:02.391+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:02.391+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:02.391+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:02.392+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:02.589+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:02.589+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:02.589+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:02.590+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:02.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:02.449+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:02.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:02.449+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:02.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:02.449+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:02.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:02.450+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:03.586 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:03.213+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14635 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:03.587 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:03.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:03.587 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:03.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:03.587 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:03.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:03.587 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:03.350+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:03.587 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:03.420+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:03.587 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:03.420+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:03.588 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:03.420+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:03.588 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:03.421+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:03.615+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:03.615+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:03.615+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:03.616+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:04.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:04.307+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:04.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:04.307+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:04.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:04.307+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:04.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:04.308+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:04.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:04.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:04.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:04.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:04.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:04.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:04.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:04.382+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:04.580+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:04.580+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:04.580+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:04.581+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:05.587 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:05.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:05.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:05.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:05.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:05.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:05.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:05.310+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:05.588 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:05.359+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:05.588 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:05.359+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:05.588 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:05.359+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:05.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:05.360+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:05.586+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:05.586+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:05.586+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:05.587+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:06.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:06.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:06.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:06.313+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:06.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:06.406+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:06.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:06.406+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:06.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:06.406+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:06.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:06.407+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:06.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:06.548+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:06.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:06.548+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:06.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:06.548+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:06.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:06.549+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:07.593 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:07.288+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:07.594 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:07.288+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:07.594 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:07.288+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:07.594 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:07.288+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:07.594 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:07.371+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:07.594 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:07.371+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:07.595 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:07.371+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:07.595 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:07.372+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:07.592+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:07.592+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:07.592+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:07.593+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:08.581 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:08.215+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14723 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:08.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:08.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:08.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:08.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:08.581 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:08.316+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:08.582 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:08.317+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:08.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:08.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:08.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:08.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:08.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:08.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:08.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:08.356+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:08.579+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:08.579+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:08.579+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:08.580+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:09.582 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:09.294+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:09.583 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:09.294+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:09.583 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:09.294+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:09.583 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:09.295+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:09.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:09.357+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:09.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:09.357+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:09.584 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:09.357+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:09.584 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:09.358+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:09.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:09.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:09.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:09.582+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:10.561 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:10.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:10.561 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:10.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:10.562 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:10.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:10.562 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:10.310+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:10.562 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:10.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:10.562 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:10.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:10.563 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:10.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:10.563 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:10.404+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:10.560+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:10.560+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:10.560+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:10.560+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:11.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:11.299+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:11.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:11.299+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:11.589 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:11.299+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:11.591 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:11.300+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:11.592 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:11.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:11.592 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:11.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:11.592 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:11.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:11.593 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:11.355+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:11.587+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:11.587+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:11.587+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:11.587+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:12.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:12.303+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:12.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:12.303+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:12.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:12.303+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:12.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:12.304+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:12.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:12.374+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:12.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:12.374+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:12.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:12.374+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:12.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:12.375+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:12.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:12.613+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:12.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:12.613+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:12.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:12.613+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:12.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:12.614+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:13.587 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:13.216+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14811 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:13.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:13.294+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:13.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:13.294+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:13.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:13.294+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:13.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:13.295+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:13.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:13.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:13.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:13.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:13.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:13.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:13.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:13.339+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:13.635+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:13.635+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:13.635+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:13.636+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:14.594 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:14.335+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:14.595 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:14.335+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:14.595 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:14.335+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:14.595 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:14.336+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:14.595 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:14.270+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:14.595 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:14.270+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:14.595 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:14.270+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:14.596 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:14.271+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:14.593+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:14.593+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:14.593+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:14.594+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:15.567 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:15.301+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:15.568 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:15.301+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:15.568 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:15.301+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:15.568 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:15.302+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:15.568 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:15.293+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:15.568 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:15.293+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:15.568 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:15.293+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:15.569 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:15.293+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:15.566+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:15.566+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:15.566+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:15.567+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:16.292+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:16.292+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:16.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:16.292+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:16.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:16.293+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:16.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:16.268+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:16.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:16.268+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:16.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:16.268+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:16.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:16.268+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:16.608+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:16.608+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:16.608+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:16.609+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:17.289+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:17.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:17.289+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:17.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:17.289+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:17.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:17.289+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:17.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:17.256+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:17.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:17.256+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:17.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:17.256+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:17.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:17.257+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:18.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:17.607+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:18.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:17.607+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:18.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:17.607+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:18.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:17.609+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:18.573 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:18.218+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14898 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:18.573 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:18.291+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:18.573 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:18.291+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:18.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:18.291+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:18.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:18.291+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:18.574 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:18.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:18.574 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:18.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:18.574 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:18.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:18.574 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:18.310+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:18.572+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:18.572+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:18.572+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:18.573+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:19.291+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:19.291+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:19.291+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:19.292+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:19.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:19.341+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:19.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:19.341+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:19.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:19.341+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:19.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:19.342+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:19.618+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:19.618+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:19.618+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:19.619+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:20.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:20.275+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:20.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:20.275+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:20.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:20.275+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:20.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:20.276+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:20.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:20.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:20.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:20.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:20.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:20.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:20.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:20.361+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:20.630+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:20.630+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:20.630+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:20.631+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:21.269+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:21.269+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:21.269+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:21.270+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:21.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:21.370+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:21.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:21.370+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:21.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:21.370+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:21.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:21.371+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:21.657+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:21.657+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:21.657+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:22.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:21.658+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:22.290+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:22.290+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:22.290+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:22.291+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:22.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:22.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:22.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:22.329+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:22.640+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:22.640+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:22.640+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:22.640+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:23.588 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:23.219+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 14987 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:23.589 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:23.269+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:23.589 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:23.269+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:23.589 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:23.269+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:23.589 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:23.270+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:23.589 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:23.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:23.590 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:23.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:23.590 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:23.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:23.590 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:23.365+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:23.677+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:23.677+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:23.677+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:23.677+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:24.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:24.326+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:24.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:24.326+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:24.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:24.326+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:24.327+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:24.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:24.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:24.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:24.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:24.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:24.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:24.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:24.309+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:24.690+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:24.690+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:24.690+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:24.690+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:25.319+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:25.319+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:25.319+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:25.319+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:25.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:25.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:25.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:25.313+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:26.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:25.679+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:26.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:25.679+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:26.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:25.679+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:26.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:25.680+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:26.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:26.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:26.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:26.365+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:26.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:26.292+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:26.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:26.292+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:26.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:26.292+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:26.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:26.292+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:26.698+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:26.698+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:26.698+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:26.699+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:27.268+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:27.268+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:27.268+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:27.269+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:27.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:27.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:27.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:27.399+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:27.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:27.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:27.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:27.737+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:28.607 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:28.221+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 15075 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:28.608 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:28.420+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:28.608 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:28.420+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:28.608 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:28.420+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:28.608 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:28.421+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:28.609 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:28.260+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:28.609 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:28.260+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:28.609 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:28.260+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:28.609 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:28.261+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:28.753+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:28.753+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:28.753+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:28.754+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:29.377+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:29.377+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:29.377+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:29.378+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:29.262+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:29.262+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:29.262+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:29.263+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:29.768+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:29.768+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:29.768+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:29.769+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:30.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:30.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:30.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:30.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:30.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:30.313+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:30.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:30.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:30.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:30.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:30.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:30.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:30.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:30.381+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:31.036 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:30.744+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:31.036 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:30.744+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:31.036 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:30.744+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:31.037 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:30.745+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:31.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:31.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:31.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:31.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:31.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:31.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:31.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:31.430+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:31.346+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:31.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:31.346+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:31.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:31.346+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:31.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:31.347+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:31.782+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:31.782+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:31.782+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:31.783+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:32.338+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:32.338+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:32.338+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:32.339+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:32.438+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:32.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:32.438+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:32.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:32.438+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:32.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:32.439+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:32.803+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:32.803+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:32.803+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:32.804+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:33.438 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:33.223+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 15164 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:33.439 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:33.331+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:33.439 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:33.331+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:33.439 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:33.331+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:33.439 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:33.332+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:33.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:33.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:33.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:33.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:33.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:33.437+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:33.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:33.438+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:34.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:33.805+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:34.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:33.805+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:34.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:33.805+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:34.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:33.805+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:34.480 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:34.297+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:34.481 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:34.297+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:34.481 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:34.297+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:34.481 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:34.298+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:34.779 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:34.479+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:34.779 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:34.479+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:34.780 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:34.479+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:34.780 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:34.480+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:35.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:34.778+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:35.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:34.778+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:35.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:34.778+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:35.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:34.779+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:35.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:35.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:35.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:35.309+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:35.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:35.310+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:35.486+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:35.486+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:35.486+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:35.487+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:35.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:35.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:35.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:35.813+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:36.293+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:36.293+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:36.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:36.293+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:36.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:36.294+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:36.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:36.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:36.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:36.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:36.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:36.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:36.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:36.476+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:37.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:36.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:37.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:36.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:37.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:36.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:37.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:36.852+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:37.286+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:37.286+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:37.286+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:37.286+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:37.448+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:37.448+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:37.448+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:37.449+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:38.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:37.845+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:38.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:37.845+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:38.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:37.845+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:38.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:37.846+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:38.492 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:38.224+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 15250 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:38.492 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:38.282+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:38.492 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:38.282+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:38.492 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:38.282+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:38.493 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:38.283+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:38.845 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:38.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:38.846 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:38.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:38.846 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:38.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:38.846 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:38.491+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:39.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:38.844+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:39.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:38.844+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:39.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:38.844+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:39.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:38.845+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:39.537 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:39.263+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:39.538 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:39.263+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:39.538 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:39.263+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:39.538 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:39.264+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:39.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:39.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:39.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:39.537+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:40.157 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:39.863+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:40.157 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:39.863+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:40.158 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:39.863+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:40.158 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:39.864+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:40.565 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:40.286+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:40.566 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:40.286+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:40.566 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:40.286+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:40.566 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:40.287+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:40.841 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:40.563+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:40.842 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:40.563+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:40.842 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:40.563+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:40.842 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:40.564+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:41.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:40.839+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:41.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:40.839+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:41.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:40.839+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:41.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:40.839+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:41.546 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:41.265+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:41.547 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:41.265+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:41.547 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:41.265+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:41.547 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:41.266+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:41.545+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:41.545+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:41.545+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:41.545+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:42.273 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:41.889+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:42.273 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:41.889+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:42.273 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:41.889+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:42.273 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:41.889+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:42.555 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:42.270+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:42.556 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:42.270+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:42.556 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:42.270+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:42.556 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:42.272+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:42.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:42.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:42.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:42.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:42.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:42.553+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:42.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:42.554+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:43.227 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:42.857+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:43.227 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:42.857+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:43.227 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:42.857+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:43.228 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:42.858+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:43.508 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:43.225+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 15339 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:43.508 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:43.260+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:43.509 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:43.260+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:43.509 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:43.260+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:43.509 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:43.260+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:43.507+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:43.507+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:43.507+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:43.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:43.507+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:44.242 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:43.888+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:44.242 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:43.888+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:44.243 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:43.888+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:44.243 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:43.888+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:44.508 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:44.241+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:44.508 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:44.241+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:44.508 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:44.241+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:44.508 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:44.242+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:44.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:44.506+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:44.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:44.506+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:44.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:44.506+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:44.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:44.507+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:45.285 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:44.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:45.285 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:44.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:45.285 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:44.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:45.285 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:44.875+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:45.480+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:45.480+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:45.480+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:45.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:45.481+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:45.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:45.284+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:45.284+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:45.284+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:45.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:45.284+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:46.309 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:45.863+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:46.310 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:45.863+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:46.310 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:45.863+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:46.310 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:45.864+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:46.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:46.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:46.308+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:46.309+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:46.491+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:46.491+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:46.491+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:46.492+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:46.828+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:46.828+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:46.828+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:46.828+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:47.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:47.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:47.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:47.330+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:47.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:47.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:47.539+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:47.540+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:47.824+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:47.824+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:47.824+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:47.824+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:48.584 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:48.227+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 15425 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:48.585 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:48.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:48.585 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:48.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:48.585 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:48.340+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:48.585 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:48.340+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:48.788+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:48.788+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:48.788+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:48.789+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:48.583+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:48.583+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:48.583+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:48.584+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:49.339+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:49.339+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:49.339+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:49.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:49.339+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:49.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:49.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:49.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:49.576+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:49.768+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:49.768+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:49.768+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:49.769+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:50.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:50.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:50.349+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:50.349+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:50.748+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:50.748+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:50.748+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:50.749+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:51.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:50.609+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:51.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:50.609+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:51.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:50.609+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:51.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:50.609+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:51.303+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:51.303+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:51.303+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:51.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:51.304+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:52.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:51.738+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:52.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:51.738+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:52.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:51.738+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:52.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:51.739+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:52.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:51.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:52.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:51.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:52.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:51.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:52.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:51.650+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:52.331+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:52.331+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:52.331+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:52.332+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:52.732+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:52.732+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:52.732+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:52.733+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:53.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:52.692+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:53.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:52.692+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:53.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:52.692+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:53.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:52.693+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:53.590 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:53.229+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 15515 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:53.590 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:53.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:53.590 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:53.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:53.591 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:53.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:53.591 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:53.358+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:53.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:53.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:53.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:53.738+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:53.689+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:53.689+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:53.689+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:53.690+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:54.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:54.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:54.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:54.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:54.344+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:55.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:54.736+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:55.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:54.736+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:55.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:54.736+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:55.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:54.737+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:55.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:54.679+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:55.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:54.679+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:55.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:54.679+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:55.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:54.680+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:55.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:55.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:55.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:55.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:55.344+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:55.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:55.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:55.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:55.722+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:55.722+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:56.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:55.722+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:56.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:55.722+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:56.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:55.722+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:56.299+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:56.299+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:56.299+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:56.300+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:56.719+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:56.719+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:56.719+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:56.720+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:57.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:56.713+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:57.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:56.713+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:57.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:56.713+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:57.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:56.714+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:57.339+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:57.339+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:57.339+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:57.340+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:58.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:57.680+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:58.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:57.680+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:58.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:57.680+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:58.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:57.681+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:58.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:57.760+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:58.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:57.760+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:58.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:57.760+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:58.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:57.761+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:58.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:34:58.231+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 15601 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:34:58.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:58.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:58.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:58.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:58.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:58.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:58.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:58.346+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:59.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:58.638+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:59.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:58.638+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:59.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:58.638+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:34:59.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:58.639+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:58.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:58.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:58.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:34:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:58.729+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:34:59.630 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:59.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:59.630 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:59.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:59.630 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:59.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:34:59.630 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:34:59.358+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:59.629+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:59.629+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:59.629+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:00.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:34:59.629+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:59.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:59.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:59.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:34:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:34:59.703+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:00.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:00.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:00.405+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:00.406+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:01.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:00.741+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:01.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:00.741+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:01.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:00.741+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:01.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:00.742+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:01.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:00.653+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:01.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:00.653+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:01.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:00.653+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:01.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:00.654+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:01.693 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:01.620+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:01.694 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:01.620+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:01.694 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:01.620+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:01.694 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:01.621+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:01.694 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:01.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:01.694 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:01.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:01.695 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:01.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:01.695 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:01.411+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:01.726+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:01.726+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:01.726+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:01.727+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:02.715 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:02.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:02.715 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:02.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:02.715 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:02.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:02.715 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:02.441+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:02.716 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:02.634+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:02.716 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:02.634+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:02.716 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:02.634+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:02.716 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:02.635+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:02.713+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:02.713+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:02.713+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:02.714+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:03.357 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:03.232+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 15691 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:03.634 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:03.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:03.634 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:03.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:03.634 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:03.407+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:03.634 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:03.408+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:03.632+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:03.632+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:03.632+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:03.633+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:04.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:03.711+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:04.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:03.711+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:04.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:03.711+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:04.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:03.712+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:04.688 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:04.615+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:04.688 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:04.615+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:04.688 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:04.615+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:04.689 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:04.616+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:04.689 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:04.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:04.689 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:04.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:04.689 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:04.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:04.689 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:04.419+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:04.687+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:04.687+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:04.687+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:04.687+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:05.459+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:05.459+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:05.459+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:05.460+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:05.657+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:05.657+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:05.657+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:05.658+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:05.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:05.656+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:05.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:05.656+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:05.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:05.656+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:05.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:05.656+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:06.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:06.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:06.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:06.469+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:06.640+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:06.640+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:06.640+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:06.640+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:06.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:06.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:06.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:06.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:06.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:06.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:06.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:06.650+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:07.700 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:07.613+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:07.700 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:07.614+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:07.700 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:07.614+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:07.700 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:07.614+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:07.701 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:07.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:07.701 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:07.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:07.701 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:07.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:07.701 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:07.440+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:07.698+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:07.698+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:07.698+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:07.699+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:08.596 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:08.234+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 15778 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:08.597 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:08.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:08.597 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:08.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:08.597 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:08.467+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:08.597 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:08.468+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:08.595+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:08.595+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:08.595+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:08.596+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:08.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:08.737+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:08.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:08.737+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:08.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:08.737+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:08.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:08.738+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:09.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:09.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:09.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:09.469+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:09.638+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:09.638+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:09.638+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:09.638+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:09.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:09.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:09.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:09.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:09.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:09.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:09.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:09.703+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:10.680 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:10.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:10.680 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:10.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:10.680 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:10.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:10.681 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:10.428+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:10.681 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:10.603+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:10.681 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:10.603+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:10.681 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:10.603+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:10.681 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:10.604+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:10.972 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:10.679+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:10.973 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:10.679+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:10.973 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:10.679+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:10.973 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:10.680+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:11.728 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:11.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:11.729 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:11.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:11.729 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:11.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:11.729 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:11.464+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:11.729 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:11.554+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:11.729 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:11.554+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:11.729 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:11.554+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:11.730 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:11.555+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:11.727+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:11.727+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:11.727+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:11.728+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:12.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:12.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:12.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:12.503+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:12.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:12.553+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:12.553+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:12.553+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:12.554+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:12.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:12.682+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:12.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:12.682+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:12.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:12.682+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:12.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:12.682+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:13.516 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:13.236+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 15868 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:13.537+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:13.537+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:13.537+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:13.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:13.538+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:13.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:13.665+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:13.665+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:13.665+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:13.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:13.666+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:13.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:13.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:13.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:13.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:13.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:13.516+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:13.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:13.516+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:14.512+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:14.512+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:14.512+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:14.513+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:14.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:14.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:14.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:14.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:14.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:14.703+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:14.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:14.703+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:14.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:14.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:14.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:14.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:14.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:14.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:14.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:14.526+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:15.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:15.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:15.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:15.530+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:15.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:15.530+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:15.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:15.530+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:15.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:15.530+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:15.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:15.531+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:15.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:15.694+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:15.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:15.694+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:15.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:15.694+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:15.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:15.695+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:16.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:16.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:16.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:16.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:16.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:16.547+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:16.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:16.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:16.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:16.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:16.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:16.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:16.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:16.484+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:16.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:16.719+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:16.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:16.719+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:16.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:16.719+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:16.859 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:16.719+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:17.744 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:17.578+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:17.744 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:17.578+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:17.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:17.578+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:17.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:17.579+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:17.745 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:17.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:17.746 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:17.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:17.746 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:17.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:17.746 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:17.469+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:17.742+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:17.742+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:17.742+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:17.743+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:18.493 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:18.237+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 15954 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:18.492+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:18.492+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:18.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:18.492+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:18.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:18.493+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:18.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:18.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:18.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:18.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:18.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:18.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:18.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:18.580+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:18.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:18.725+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:18.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:18.725+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:18.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:18.725+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:18.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:18.726+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:19.724 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:19.612+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:19.724 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:19.612+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:19.725 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:19.612+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:19.725 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:19.613+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:19.725 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:19.444+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:19.726 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:19.444+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:19.726 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:19.444+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:19.726 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:19.444+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:19.721+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:19.721+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:19.721+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:19.722+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:20.673+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:20.673+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:20.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:20.673+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:20.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:20.673+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:20.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:20.568+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:20.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:20.568+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:20.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:20.568+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:20.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:20.569+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:20.858 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:20.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:20.858 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:20.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:20.858 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:20.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:20.859 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:20.460+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:21.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:21.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:21.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:21.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:21.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:21.528+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:21.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:21.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:21.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:21.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:21.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:21.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:21.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:21.508+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:21.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:21.709+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:21.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:21.709+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:21.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:21.709+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:21.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:21.710+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:22.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:22.486+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:22.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:22.486+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:22.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:22.486+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:22.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:22.487+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:22.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:22.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:22.705+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:22.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:22.706+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:22.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:22.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:22.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:22.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:22.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:22.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:22.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:22.490+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:23.508 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:23.239+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 16043 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:23.509 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:23.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:23.509 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:23.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:23.509 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:23.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:23.509 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:23.484+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:23.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:23.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:23.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:23.508+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:23.681+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:23.681+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:23.681+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:23.682+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:24.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:24.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:24.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:24.473+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:24.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:24.476+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:24.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:24.476+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:24.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:24.476+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:24.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:24.477+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:24.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:24.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:24.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:24.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:24.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:24.669+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:24.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:24.669+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:25.716 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:25.452+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:25.717 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:25.452+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:25.717 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:25.452+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:25.717 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:25.453+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:25.717 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:25.455+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:25.717 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:25.455+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:25.718 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:25.455+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:25.718 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:25.456+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:25.714+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:25.714+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:25.714+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:25.715+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:26.466+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:26.466+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:26.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:26.466+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:26.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:26.467+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:26.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:26.428+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:26.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:26.428+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:26.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:26.428+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:26.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:26.429+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:26.747+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:26.747+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:26.747+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:26.748+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:27.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:27.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:27.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:27.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:27.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:27.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:27.797 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:27.434+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:27.797 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:27.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:27.797 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:27.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:27.798 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:27.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:27.798 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:27.427+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:28.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:27.793+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:28.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:27.793+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:28.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:27.793+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:28.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:27.794+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:28.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:28.240+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 16128 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:28.422+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:28.422+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:28.422+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:28.423+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:28.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:28.455+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:28.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:28.455+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:28.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:28.455+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:28.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:28.456+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:28.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:28.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:28.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:28.764+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:29.780 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:29.503+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:29.780 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:29.503+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:29.780 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:29.503+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:29.780 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:29.504+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:29.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:29.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:29.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:29.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:29.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:29.414+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:29.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:29.414+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:29.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:29.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:29.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:29.779+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:30.538 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:30.394+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:30.539 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:30.394+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:30.539 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:30.394+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:30.539 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:30.395+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:30.789 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:30.537+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:30.789 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:30.537+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:30.789 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:30.537+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:30.789 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:30.538+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:30.788+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:30.788+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:30.788+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:31.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:30.789+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:31.696 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:31.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:31.696 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:31.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:31.697 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:31.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:31.697 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:31.439+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:31.697 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:31.525+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:31.697 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:31.525+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:31.697 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:31.525+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:31.698 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:31.526+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:31.760+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:31.760+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:31.760+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:31.761+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:32.789 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:32.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:32.789 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:32.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:32.789 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:32.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:32.790 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:32.528+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:32.790 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:32.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:32.790 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:32.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:32.790 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:32.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:32.790 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:32.486+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:32.788+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:32.788+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:32.788+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:32.789+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:33.368 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:33.243+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 16218 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:33.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:33.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:33.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:33.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:33.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:33.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:33.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:33.535+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:33.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:33.486+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:33.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:33.487+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:33.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:33.487+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:33.806 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:33.487+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:33.803+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:33.803+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:33.803+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:33.804+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:34.819 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:34.450+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:34.819 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:34.450+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:34.819 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:34.450+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:34.820 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:34.451+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:34.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:34.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:34.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:34.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:34.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:34.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:34.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:34.527+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:34.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:34.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:34.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:34.818+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:35.774 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:35.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:35.774 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:35.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:35.774 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:35.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:35.774 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:35.514+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:35.775 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:35.428+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:35.775 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:35.428+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:35.775 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:35.428+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:35.775 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:35.428+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:36.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:35.772+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:36.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:35.772+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:36.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:35.772+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:36.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:35.774+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:36.738 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:36.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:36.738 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:36.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:36.738 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:36.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:36.738 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:36.529+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:36.739 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:36.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:36.739 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:36.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:36.739 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:36.448+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:36.739 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:36.448+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:37.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:36.737+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:37.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:36.737+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:37.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:36.737+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:37.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:36.738+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:37.755 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:37.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:37.756 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:37.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:37.756 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:37.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:37.756 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:37.560+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:37.756 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:37.471+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:37.756 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:37.471+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:37.757 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:37.471+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:37.757 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:37.471+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:38.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:37.754+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:38.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:37.754+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:38.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:37.754+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:38.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:37.755+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:38.533 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:38.244+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 16304 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:38.534 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:38.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:38.534 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:38.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:38.534 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:38.427+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:38.534 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:38.427+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:38.532+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:38.532+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:38.532+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:38.533+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:38.761+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:38.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:38.761+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:38.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:38.761+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:38.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:38.762+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:39.744 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:39.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:39.745 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:39.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:39.745 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:39.452+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:39.745 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:39.453+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:39.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:39.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:39.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:39.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:39.746 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:39.535+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:39.746 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:39.536+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:39.743+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:39.743+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:39.743+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:39.744+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:40.758 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:40.491+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:40.758 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:40.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:40.758 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:40.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:40.758 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:40.493+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:40.759 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:40.441+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:40.759 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:40.441+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:40.759 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:40.441+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:40.759 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:40.442+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:41.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:40.756+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:41.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:40.756+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:41.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:40.756+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:41.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:40.757+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:41.776 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:41.464+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:41.776 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:41.464+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:41.776 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:41.464+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:41.776 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:41.465+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:41.777 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:41.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:41.777 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:41.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:41.777 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:41.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:41.777 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:41.471+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:42.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:41.774+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:42.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:41.774+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:42.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:41.774+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:42.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:41.776+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:42.514+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:42.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:42.514+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:42.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:42.514+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:42.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:42.514+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:42.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:42.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:42.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:42.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:42.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:42.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:42.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:42.508+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:42.823+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:42.823+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:43.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:42.823+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:43.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:42.824+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:43.508 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:43.246+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 16392 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:43.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:43.544+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:43.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:43.544+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:43.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:43.544+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:43.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:43.545+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:43.825 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:43.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:43.825 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:43.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:43.825 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:43.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:43.825 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:43.508+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:44.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:43.822+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:44.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:43.822+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:44.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:43.822+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:44.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:43.823+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:44.764 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:44.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:44.764 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:44.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:44.765 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:44.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:44.765 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:44.544+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:44.765 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:44.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:44.765 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:44.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:44.765 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:44.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:44.765 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:44.462+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:45.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:44.827+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:45.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:44.827+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:45.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:44.827+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:45.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:44.828+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:45.806 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:45.534+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:45.806 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:45.534+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:45.806 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:45.534+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:45.806 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:45.535+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:45.807 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:45.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:45.807 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:45.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:45.807 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:45.477+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:45.807 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:45.477+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:45.805+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:45.805+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:45.805+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:45.805+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:46.768 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:46.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:46.768 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:46.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:46.769 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:46.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:46.769 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:46.458+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:46.769 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:46.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:46.769 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:46.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:46.769 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:46.585+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:46.769 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:46.586+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:47.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:46.767+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:47.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:46.767+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:47.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:46.767+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:47.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:46.768+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:47.784 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:47.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:47.785 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:47.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:47.785 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:47.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:47.785 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:47.499+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:47.785 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:47.616+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:47.785 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:47.616+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:47.786 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:47.616+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:47.786 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:47.617+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:47.783+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:47.783+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:47.783+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:47.784+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:48.583 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:48.248+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 16479 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:48.584 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:48.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:48.584 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:48.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:48.584 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:48.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:48.584 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:48.471+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:48.789+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:48.789+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:48.789+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:48.790+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:48.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:48.582+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:48.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:48.582+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:48.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:48.582+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:48.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:48.583+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:49.790 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:49.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:49.790 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:49.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:49.790 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:49.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:49.791 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:49.461+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:49.791 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:49.614+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:49.791 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:49.614+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:49.791 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:49.614+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:49.791 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:49.614+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:50.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:49.789+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:50.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:49.789+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:50.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:49.789+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:50.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:49.790+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:50.799 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:50.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:50.799 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:50.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:50.799 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:50.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:50.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:50.501+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:50.800 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:50.631+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:50.800 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:50.631+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:50.800 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:50.631+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:50.800 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:50.632+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:50.798+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:50.798+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:50.798+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:50.799+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:51.771 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:51.631+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:51.771 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:51.631+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:51.771 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:51.631+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:51.771 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:51.631+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:51.771 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:51.769+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:51.772 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:51.769+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:51.772 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:51.769+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:51.772 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:51.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:51.772 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:51.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:51.772 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:51.485+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:51.772 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:51.485+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:52.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:51.770+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:52.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:52.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:52.520+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:52.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:52.520+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:52.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:52.675+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:52.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:52.675+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:52.806 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:52.675+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:52.806 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:52.676+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:52.810+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:52.810+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:52.810+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:52.811+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:53.546 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:53.250+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 16569 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:53.815 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:53.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:53.872 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:53.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:53.872 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:53.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:53.872 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:53.546+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:53.873 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:53.723+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:53.873 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:53.723+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:53.873 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:53.723+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:53.873 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:53.724+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:53.814+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:53.814+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:53.814+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:53.814+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:54.801 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:54.702+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:54.801 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:54.702+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:54.801 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:54.702+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:54.802 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:54.703+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:54.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:54.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:54.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:54.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:54.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:54.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:54.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:54.518+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:54.800+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:54.800+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:54.800+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:54.801+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:55.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:55.739+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:55.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:55.739+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:55.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:55.739+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:55.740+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:55.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:55.761+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:55.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:55.761+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:55.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:55.761+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:55.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:55.762+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:55.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:55.544+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:55.858 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:55.544+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:55.858 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:55.544+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:55.858 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:55.545+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:56.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:56.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:56.560+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:56.561+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:56.760+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:56.760+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:56.760+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:56.761+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:56.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:56.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:56.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:56.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:56.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:56.739+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:56.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:56.740+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:57.524+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:57.524+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:57.524+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:57.525+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:57.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:57.748+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:57.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:57.748+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:57.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:57.748+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:57.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:57.749+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:57.751+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:57.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:57.751+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:57.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:57.751+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:57.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:57.752+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:58.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:35:58.251+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 16655 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:35:58.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:58.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:58.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:58.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:58.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:58.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:58.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:58.482+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:59.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:58.792+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:59.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:58.792+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:59.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:58.792+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:35:59.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:58.792+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:58.724+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:58.724+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:58.724+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:35:59.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:58.725+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:35:59.692 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:59.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:59.692 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:59.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:59.693 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:59.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:35:59.693 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:35:59.435+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:59.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:59.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:59.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:35:59.830+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:59.691+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:59.691+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:59.691+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:00.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:35:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:35:59.692+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:00.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:00.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:00.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:00.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:00.412+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:01.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:00.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:01.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:00.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:01.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:00.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:01.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:00.874+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:01.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:00.699+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:01.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:00.699+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:01.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:00.699+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:01.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:00.700+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:01.669 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:01.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:01.669 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:01.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:01.669 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:01.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:01.669 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:01.375+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:02.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:01.888+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:02.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:01.888+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:02.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:01.888+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:02.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:01.889+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:02.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:01.667+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:02.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:01.667+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:02.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:01.667+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:02.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:01.668+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:02.689 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:02.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:02.689 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:02.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:02.689 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:02.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:02.689 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:02.385+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:02.687+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:02.687+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:02.687+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:02.688+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:02.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:03.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:02.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:03.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:02.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:03.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:02.875+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:03.377 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:03.253+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 16744 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:03.731 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:03.419+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:03.731 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:03.419+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:03.731 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:03.419+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:03.731 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:03.420+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:03.840+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:03.840+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:03.840+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:03.841+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:04.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:03.729+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:04.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:03.730+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:04.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:03.730+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:04.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:03.730+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:04.780 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:04.408+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:04.780 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:04.408+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:04.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:04.408+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:04.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:04.409+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:05.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:04.825+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:05.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:04.825+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:05.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:04.825+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:05.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:04.826+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:05.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:04.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:05.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:04.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:05.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:04.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:05.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:04.780+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:05.777 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:05.403+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:05.777 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:05.403+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:05.777 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:05.403+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:05.777 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:05.404+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:06.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:05.804+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:06.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:05.804+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:06.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:05.804+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:06.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:05.805+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:06.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:05.776+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:06.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:05.776+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:06.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:05.776+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:06.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:05.777+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:06.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:06.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:06.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:06.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:06.397+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:06.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:06.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:06.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:06.852+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:07.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:06.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:07.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:06.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:07.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:06.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:07.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:06.818+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:07.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:07.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:07.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:07.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:07.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:07.402+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:07.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:07.403+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:08.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:07.807+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:08.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:07.807+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:08.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:07.807+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:08.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:07.808+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:08.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:07.798+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:08.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:07.798+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:08.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:07.798+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:08.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:07.799+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:08.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:08.255+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 16831 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:08.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:08.385+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:08.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:08.385+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:08.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:08.385+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:08.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:08.386+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:08.764+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:08.764+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:08.764+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:08.764+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:09.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:08.822+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:09.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:08.822+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:09.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:08.822+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:09.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:08.822+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:09.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:09.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:09.352+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:09.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:09.353+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:09.855+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:09.855+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:09.855+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:09.856+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:09.801+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:10.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:09.801+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:10.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:09.801+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:10.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:09.802+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:10.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:10.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:10.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:10.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:10.381+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:10.840+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:10.840+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:10.840+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:10.841+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:11.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:10.834+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:11.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:10.834+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:11.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:10.834+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:11.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:10.835+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:11.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:11.332+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:11.332+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:11.332+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:11.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:11.333+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:11.872+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:11.872+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:11.872+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:11.873+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:12.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:11.877+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:12.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:11.877+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:12.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:11.877+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:12.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:11.877+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:12.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:12.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:12.381+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:12.382+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:13.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:12.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:13.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:12.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:13.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:12.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:13.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:12.830+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:12.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:12.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:12.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:12.843+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:13.595 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:13.256+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 16922 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:13.595 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:13.380+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:13.595 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:13.380+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:13.595 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:13.380+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:13.596 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:13.380+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:13.884+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:13.884+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:13.884+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:14.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:13.885+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:13.850+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:13.850+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:13.850+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:13.850+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:14.829 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:14.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:14.829 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:14.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:14.830 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:14.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:14.830 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:14.411+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:14.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:14.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:14.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:14.879+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:15.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:14.828+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:15.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:14.828+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:15.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:14.828+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:15.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:14.829+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:15.809 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:15.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:15.809 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:15.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:15.809 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:15.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:15.809 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:15.431+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:15.807+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:15.807+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:16.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:15.807+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:16.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:15.809+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:15.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:15.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:16.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:15.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:16.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:15.923+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:16.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:16.454+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:16.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:16.454+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:16.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:16.454+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:16.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:16.454+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:17.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:16.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:17.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:16.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:16.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:16.879+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:17.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:16.798+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:17.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:16.798+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:17.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:16.798+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:17.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:16.799+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:17.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:17.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:17.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:17.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:17.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:17.504+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:17.823 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:17.505+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:17.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:17.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:17.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:17.836+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:18.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:17.822+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:18.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:17.822+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:18.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:17.822+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:18.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:17.823+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:18.553 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:18.258+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17009 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:18.827 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:18.552+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:18.827 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:18.552+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:18.827 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:18.552+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:18.827 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:18.553+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:19.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:18.826+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:19.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:18.826+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:19.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:18.826+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:19.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:18.827+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:19.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:18.858+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:19.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:18.858+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:19.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:18.858+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:19.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:18.859+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:19.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:19.578+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:19.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:19.578+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:19.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:19.578+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:19.853 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:19.579+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:19.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:19.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:20.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:19.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:20.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:19.852+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:20.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:19.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:20.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:19.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:20.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:19.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:20.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:19.897+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:20.830 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:20.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:20.830 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:20.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:20.831 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:20.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:20.831 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:20.535+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:20.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:20.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:20.945+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:20.945+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:21.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:20.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:21.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:20.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:21.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:20.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:21.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:20.830+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:21.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:21.548+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:21.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:21.548+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:21.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:21.548+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:21.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:21.549+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:21.802+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:22.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:21.802+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:22.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:21.802+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:22.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:21.803+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:21.988+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:21.988+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:21.988+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:22.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:21.989+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:22.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:22.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:22.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:22.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:22.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:22.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:22.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:22.583+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:22.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:22.781+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:22.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:22.781+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:22.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:22.781+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:22.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:22.782+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:23.283 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:23.260+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17099 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:23.284 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:23.021+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:23.284 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:23.021+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:23.284 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:23.021+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:23.284 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:23.022+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:23.815 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:23.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:23.815 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:23.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:23.815 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:23.543+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:23.815 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:23.544+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:24.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:23.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:24.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:23.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:24.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:23.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:24.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:23.814+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:24.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:24.044+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:24.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:24.044+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:24.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:24.044+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:24.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:24.045+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:24.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:24.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:24.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:24.581+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:24.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:24.800+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:24.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:24.800+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:24.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:24.800+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:24.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:24.801+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:25.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:25.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:25.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:25.027+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:25.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:25.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:25.566+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:25.567+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:25.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:25.797+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:25.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:25.797+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:25.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:25.797+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:25.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:25.797+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:26.031+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:26.031+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:26.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:26.031+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:26.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:26.033+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:26.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:26.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:26.801 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:26.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:26.801 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:26.534+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:26.801 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:26.534+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:26.799+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:26.799+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:26.799+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:26.800+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:27.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:26.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:27.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:26.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:27.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:26.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:27.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:26.983+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:27.814 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:27.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:27.814 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:27.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:27.814 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:27.511+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:27.814 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:27.511+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:27.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:27.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:27.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:27.814+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:28.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:27.961+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:28.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:27.961+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:28.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:27.961+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:28.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:27.961+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:28.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:28.262+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17186 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:28.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:28.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:28.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:28.501+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:28.989+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:28.989+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:28.989+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:28.990+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:29.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:28.830+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:29.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:28.830+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:29.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:28.830+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:29.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:28.831+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:29.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:29.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:29.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:29.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:29.506+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:30.165 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:29.863+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:30.166 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:29.863+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:30.166 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:29.863+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:30.166 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:29.864+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:30.166 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:29.975+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:30.167 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:29.975+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:30.167 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:29.975+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:30.167 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:29.976+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:30.844 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:30.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:30.844 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:30.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:30.844 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:30.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:30.844 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:30.529+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:31.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:30.871+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:31.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:30.871+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:31.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:30.871+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:31.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:30.872+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:31.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:30.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:31.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:30.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:31.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:30.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:31.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:30.942+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:31.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:31.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:31.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:31.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:31.493+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:31.918+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:31.918+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:31.918+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:31.919+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:31.976+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:31.976+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:31.976+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:31.977+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:32.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:32.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:32.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:32.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:32.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:32.528+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:33.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:33.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:33.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:33.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:33.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:33.024+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:33.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:32.895+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:33.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:32.895+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:33.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:32.895+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:33.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:32.896+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:33.386 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:33.264+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17277 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:33.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:33.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:33.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:33.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:33.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:33.573+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:33.853 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:33.573+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:34.026+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:34.026+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:34.026+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:34.027+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:34.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:33.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:34.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:33.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:34.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:33.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:34.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:33.852+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:34.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:34.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:34.528+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:34.528+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:34.891+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:34.891+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:34.891+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:34.892+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:35.045+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:35.045+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:35.045+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:35.046+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:35.532+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:35.532+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:35.532+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:35.533+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:35.919+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:35.919+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:35.919+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:35.920+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:36.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:36.049+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:36.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:36.049+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:36.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:36.049+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:36.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:36.051+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:36.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:36.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:36.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:36.490+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:36.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:36.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:36.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:36.957+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:37.099+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:37.099+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:37.099+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:37.100+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:37.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:37.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:37.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:37.461+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:38.267 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:37.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:38.267 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:37.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:38.267 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:37.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:38.267 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:37.999+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:38.268 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:38.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:38.268 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:38.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:38.268 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:38.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:38.268 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:38.080+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:38.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:38.265+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17364 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:38.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:38.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:38.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:38.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:38.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:38.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:38.428+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:39.021+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:39.021+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:39.021+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:39.022+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:39.059+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:39.059+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:39.059+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:39.060+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:39.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:39.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:39.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:39.388+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:40.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:40.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:40.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:40.036+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:40.088+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:40.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:40.088+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:40.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:40.088+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:40.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:40.089+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:40.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:40.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:40.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:40.360+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:41.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:41.024+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:41.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:41.024+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:41.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:41.024+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:41.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:41.025+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:41.341 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:41.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:41.341 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:41.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:41.341 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:41.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:41.341 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:41.055+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:41.338+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:41.338+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:41.338+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:41.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:41.340+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:42.330 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:42.020+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:42.330 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:42.020+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:42.331 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:42.020+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:42.331 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:42.020+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:42.331 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:42.059+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:42.331 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:42.059+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:42.331 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:42.059+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:42.331 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:42.059+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:42.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:42.329+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:42.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:42.329+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:42.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:42.329+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:42.588 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:42.330+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:43.132 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:43.047+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:43.132 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:43.047+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:43.133 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:43.047+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:43.133 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:43.048+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:43.133 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:43.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:43.133 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:43.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:43.133 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:43.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:43.134 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:43.024+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:43.391 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:43.267+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17454 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:43.391 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:43.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:43.392 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:43.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:43.392 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:43.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:43.392 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:43.344+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:44.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:44.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:44.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:44.033+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:44.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:44.072+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:44.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:44.072+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:44.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:44.072+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:44.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:44.073+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:44.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:44.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:44.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:44.371+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:45.025+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:45.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:45.025+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:45.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:45.025+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:45.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:45.026+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:45.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:45.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:45.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:45.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:45.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:45.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:45.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:45.069+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:45.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:45.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:45.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:45.374+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:45.998+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:45.998+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:45.998+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:46.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:45.999+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:46.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:46.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:46.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:46.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:46.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:46.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:46.091+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:46.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:46.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:46.370+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:46.371+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:47.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:47.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:47.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:47.107+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:47.035+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:47.035+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:47.035+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:47.036+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:47.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:47.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:47.386+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:47.387+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:48.280 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:48.081+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:48.281 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:48.081+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:48.281 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:48.081+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:48.281 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:48.082+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:48.281 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:48.269+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17541 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:48.282 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:48.020+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:48.282 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:48.020+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:48.282 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:48.020+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:48.282 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:48.021+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:48.420+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:48.420+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:48.420+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:48.421+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:49.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:49.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:49.078+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:49.079+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:49.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:49.057+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:49.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:49.057+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:49.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:49.057+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:49.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:49.058+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:49.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:49.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:49.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:49.442+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:50.077+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:50.077+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:50.077+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:50.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:50.078+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:50.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:50.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:50.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:50.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:50.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:50.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:50.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:50.091+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:50.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:50.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:50.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:50.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:50.477+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:51.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:51.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:51.055+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:51.056+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:51.119+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:51.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:51.119+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:51.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:51.119+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:51.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:51.120+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:51.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:51.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:51.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:51.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:51.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:51.496+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:52.348 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:52.132+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:52.349 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:52.132+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:52.349 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:52.132+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:52.349 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:52.132+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:52.349 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:52.053+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:52.350 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:52.053+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:52.350 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:52.053+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:52.350 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:52.054+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:52.493+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:52.493+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:52.493+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:52.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:52.494+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:53.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:53.271+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17632 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:53.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:53.096+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:53.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:53.096+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:53.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:53.096+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:53.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:53.097+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:53.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:53.173+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:53.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:53.173+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:53.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:53.173+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:53.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:53.174+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:53.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:53.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:53.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:53.465+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:54.048+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:54.048+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:54.048+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:54.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:54.049+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:54.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:54.144+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:54.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:54.144+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:54.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:54.144+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:54.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:54.145+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:54.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:54.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:54.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:54.461+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:55.087+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:55.087+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:55.087+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:55.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:55.088+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:55.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:55.172+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:55.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:55.172+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:55.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:55.172+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:55.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:55.172+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:55.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:55.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:55.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:55.432+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:56.211+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:56.211+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:56.211+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:56.212+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:56.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:56.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:56.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:56.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:56.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:56.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:56.057+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:56.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:56.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:56.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:56.474+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:57.043+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:57.043+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:57.043+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:57.044+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:57.250+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:57.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:57.250+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:57.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:57.250+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:57.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:57.251+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:57.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:57.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:57.425+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:57.425+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:58.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:36:58.271+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17719 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:36:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:58.084+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:58.084+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:58.084+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:58.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:58.085+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:58.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:58.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:58.218+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:58.219+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:58.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:58.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:58.379+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:58.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:58.380+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:59.404 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:59.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:59.405 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:59.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:59.405 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:59.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:36:59.405 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:36:59.266+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:59.405 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:59.128+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:59.405 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:59.128+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:59.406 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:59.128+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:36:59.406 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:36:59.129+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:36:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:59.403+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:59.403+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:59.403+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:36:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:36:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:36:59.403+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:00.308+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:00.308+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:00.308+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:00.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:00.309+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:00.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:00.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:00.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:00.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:00.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:00.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:00.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:00.366+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:00.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:00.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:00.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:00.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:00.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:00.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:00.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:00.128+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:01.347 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:01.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:01.347 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:01.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:01.348 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:01.330+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:01.348 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:01.331+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:01.348 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:01.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:01.348 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:01.095+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:01.348 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:01.095+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:01.348 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:01.095+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:01.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:01.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:01.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:01.347+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:02.375 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:02.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:02.375 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:02.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:02.375 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:02.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:02.375 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:02.107+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:02.376 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:02.312+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:02.376 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:02.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:02.376 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:02.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:02.376 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:02.313+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:02.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:02.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:02.373+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:02.374+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:03.375 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:03.274+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17808 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:03.376 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:03.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:03.376 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:03.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:03.376 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:03.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:03.376 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:03.305+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:03.376 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:03.135+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:03.377 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:03.135+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:03.377 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:03.135+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:03.377 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:03.135+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:03.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:03.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:03.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:03.375+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:04.316+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:04.316+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:04.316+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:04.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:04.317+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:04.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:04.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:04.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:04.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:04.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:04.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:04.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:04.102+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:04.422+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:04.422+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:04.422+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:04.423+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:05.452 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:05.139+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:05.453 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:05.139+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:05.453 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:05.139+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:05.453 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:05.140+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:05.453 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:05.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:05.453 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:05.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:05.453 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:05.270+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:05.453 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:05.271+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:05.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:05.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:05.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:05.452+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:06.443 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:06.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:06.443 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:06.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:06.443 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:06.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:06.443 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:06.293+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:06.443 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:06.144+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:06.444 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:06.144+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:06.444 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:06.144+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:06.444 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:06.145+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:06.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:06.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:06.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:06.442+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:07.492 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:07.276+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:07.492 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:07.276+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:07.492 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:07.276+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:07.492 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:07.277+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:07.493 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:07.168+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:07.493 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:07.168+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:07.493 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:07.168+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:07.493 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:07.169+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:07.491+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:07.491+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:07.491+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:07.491+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:08.509 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:08.276+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17894 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:08.510 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:08.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:08.510 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:08.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:08.510 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:08.313+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:08.511 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:08.314+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:08.511 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:08.160+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:08.511 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:08.160+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:08.511 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:08.160+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:08.511 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:08.160+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:08.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:08.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:08.508+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:08.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:08.509+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:09.527 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:09.298+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:09.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:09.298+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:09.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:09.298+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:09.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:09.299+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:09.528 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:09.189+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:09.528 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:09.189+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:09.529 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:09.189+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:09.529 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:09.189+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:09.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:09.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:09.526+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:09.527+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:10.500 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:10.272+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:10.501 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:10.272+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:10.501 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:10.272+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:10.501 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:10.273+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:10.501 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:10.151+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:10.502 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:10.151+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:10.502 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:10.152+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:10.502 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:10.152+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:10.780 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:10.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:10.780 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:10.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:10.780 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:10.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:10.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:10.500+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:11.459 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:11.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:11.459 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:11.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:11.459 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:11.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:11.460 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:11.310+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:11.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:11.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:11.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:11.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:11.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:11.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:11.460 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:11.111+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:11.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:11.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:11.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:11.459+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:12.274+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:12.274+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:12.274+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:12.275+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:12.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:12.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:12.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:12.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:12.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:12.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:12.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:12.103+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:12.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:12.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:12.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:12.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:12.419+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:13.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:13.278+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 17984 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:13.232+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:13.232+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:13.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:13.232+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:13.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:13.232+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:13.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:13.085+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:13.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:13.085+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:13.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:13.085+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:13.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:13.086+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:13.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:13.383+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:13.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:13.383+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:13.852 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:13.383+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:13.853 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:13.384+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:14.060+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:14.060+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:14.060+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:14.061+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:14.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:14.229+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:14.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:14.229+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:14.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:14.229+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:14.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:14.230+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:14.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:14.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:14.343+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:14.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:14.344+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:15.228+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:15.228+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:15.228+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:15.229+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:15.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:15.337+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:15.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:15.337+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:15.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:15.337+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:15.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:15.338+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:15.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:15.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:15.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:15.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:15.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:15.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:15.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:15.095+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:16.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:16.258+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:16.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:16.258+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:16.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:16.258+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:16.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:16.258+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:16.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:16.050+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:16.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:16.050+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:16.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:16.050+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:16.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:16.051+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:16.371+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:16.371+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:16.371+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:16.372+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:17.285 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:17.002+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:17.285 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:17.002+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:17.286 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:17.002+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:17.286 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:17.003+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:17.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:17.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:17.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:43.503728+0000 (oldest deadline 2024-02-10T06:27:03.503728+0000) 2024-02-10T06:37:17.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:17.397+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:17.284+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:17.284+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:17.284+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:17.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:17.284+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:18.242 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:17.956+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:18.242 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:17.956+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:18.242 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:17.956+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:18.242 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:17.956+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:18.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:18.280+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18070 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:18.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:18.434+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:18.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:18.240+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:18.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:18.240+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:18.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:18.240+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:18.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:18.241+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:19.291 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:18.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:19.291 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:18.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:19.291 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:18.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:19.291 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:19.000+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:19.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:19.401+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:19.290+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:19.290+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:19.290+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:48.995269+0000 (oldest deadline 2024-02-10T06:27:08.995269+0000) 2024-02-10T06:37:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:19.291+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:20.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:20.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:20.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:20.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:20.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:20.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:20.316 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:20.028+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:20.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:20.414+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:20.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:20.314+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:21.336 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:21.065+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:21.336 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:21.065+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:21.336 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:21.065+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:21.337 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:21.066+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:21.335+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:21.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:21.435+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:22.327+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:22.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:22.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:22.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:26:51.532441+0000 (oldest deadline 2024-02-10T06:27:11.532441+0000) 2024-02-10T06:37:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:22.104+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:22.417+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:23.309+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:23.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:23.098+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:23.356 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:23.282+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18159 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:23.412+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:24.291+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:24.068+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:24.406+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:25.249+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:25.045+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:25.374+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:26.226+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:26.037+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:26.403+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:27.273+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:27.062+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:27.434+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:28.285 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:28.029+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:28.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:28.284+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18245 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:28.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:28.404+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:28.318+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:29.302 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:29.013+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:29.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:29.301+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:29.388+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:30.286 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:30.035+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:30.364+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:30.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:30.285+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:31.322 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:31.017+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:31.388+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:31.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:31.321+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:32.273+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:32.033+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:32.417+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:32.989+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:33.379 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:33.286+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18335 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:33.379 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:33.309+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:33.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:33.377+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:34.288 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:34.018+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:34.417+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:34.287+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:35.308 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:35.033+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:35.459+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:35.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:35.307+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:36.338 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:36.048+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:36.471+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:36.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:36.337+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:37.341 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:37.081+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:37.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:37.424+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:37.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:37.340+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:38.303+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:38.355 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:38.288+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18421 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:38.061+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:38.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:38.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:38.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:38.465+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:39.325+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:39.098+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:39.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:39.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:39.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:39.491+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:40.481 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:40.360+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:40.481 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:40.134+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:40.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:40.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:40.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:40.481+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:41.093+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:41.703 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:41.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:41.703 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:41.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:41.703 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:41.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:41.704 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:41.431+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:41.704 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:41.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:41.704 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:41.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:41.704 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:41.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:41.704 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:41.374+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:42.335 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:42.072+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:42.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:42.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:42.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:42.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:42.334+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:42.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:42.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:42.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:42.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:42.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:42.471+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:42.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:42.471+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:43.313 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:43.289+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18509 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:43.313 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:43.058+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:43.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:43.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:43.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:43.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:43.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:43.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:43.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:43.339+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:43.602 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:43.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:43.602 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:43.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:43.602 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:43.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:43.602 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:43.433+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:44.328 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:44.048+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:44.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:44.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:44.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:44.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:44.327+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:44.437+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:44.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:44.437+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:44.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:44.437+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:44.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:44.437+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:45.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:45.018+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:45.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:45.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:45.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:45.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:45.370+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:45.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:45.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:45.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:45.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:45.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:45.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:45.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:45.419+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:46.340 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:45.985+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:46.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:46.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:46.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:46.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:46.340+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:46.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:46.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:46.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:46.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:46.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:46.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:46.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:46.457+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:47.000+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:47.000+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:47.000+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:47.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:47.001+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:47.443+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:47.443+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:47.443+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:47.444+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:47.360+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:47.360+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:47.360+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:47.360+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:48.293 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:48.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:48.293 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:48.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:48.293 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:48.027+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:48.293 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:48.028+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:48.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:48.291+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18596 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:48.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:48.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:48.363+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:48.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:48.364+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:48.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:48.453+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:48.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:48.453+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:48.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:48.453+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:48.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:48.454+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:49.333 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:49.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:49.333 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:49.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:49.333 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:49.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:49.333 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:49.028+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:49.332+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:49.332+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:49.332+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:49.332+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:49.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:49.491+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:49.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:49.491+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:49.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:49.491+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:49.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:49.491+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:50.334 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:50.036+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:50.335 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:50.036+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:50.335 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:50.036+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:50.335 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:50.038+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:50.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:50.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:50.333+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:50.334+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:50.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:50.459+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:50.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:50.459+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:50.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:50.459+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:50.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:50.460+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:51.066+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:51.066+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:51.066+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:51.067+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:51.704 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:51.469+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:51.704 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:51.469+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:51.704 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:51.469+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:51.704 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:51.470+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:51.704 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:51.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:51.705 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:51.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:51.705 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:51.370+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:51.705 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:51.370+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:52.098+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:52.099+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:52.099+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:52.100+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:52.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:52.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:52.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:52.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:52.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:52.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:52.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:52.329+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:52.436+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:52.436+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:52.436+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:52.437+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:53.416 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:53.293+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18685 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:53.417 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:53.129+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:53.417 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:53.129+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:53.417 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:53.129+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:53.417 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:53.130+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:53.417 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:53.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:53.417 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:53.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:53.418 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:53.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:53.418 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:53.310+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:53.789 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:53.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:53.789 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:53.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:53.789 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:53.434+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:53.789 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:53.434+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:54.471 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:54.347+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:54.472 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:54.347+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:54.472 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:54.347+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:54.472 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:54.348+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:54.472 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:54.157+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:54.472 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:54.157+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:54.472 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:54.157+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:54.473 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:54.158+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:54.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:54.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:54.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:54.471+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:55.434 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:55.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:55.434 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:55.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:55.434 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:55.387+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:55.434 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:55.387+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:55.435 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:55.152+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:55.435 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:55.152+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:55.435 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:55.152+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:55.435 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:55.153+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:55.433+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:55.433+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:55.433+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:55.434+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:56.461 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:56.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:56.461 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:56.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:56.461 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:56.373+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:56.461 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:56.374+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:56.461 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:56.138+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:56.462 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:56.138+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:56.462 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:56.138+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:56.462 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:56.138+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:56.459+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:56.459+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:56.459+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:56.461+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:57.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:57.183+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:57.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:57.183+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:57.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:57.183+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:57.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:57.184+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:57.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:57.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:57.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:57.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:57.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:57.411+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:57.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:57.412+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:57.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:57.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:57.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:57.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:57.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:57.355+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:57.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:57.355+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:58.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:37:58.295+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18772 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:37:58.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:58.406+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:58.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:58.406+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:58.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:58.406+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:58.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:58.407+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:58.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:58.186+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:58.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:58.186+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:58.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:58.186+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:58.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:58.187+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:58.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:58.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:58.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:58.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:58.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:58.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:58.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:58.385+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:59.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:59.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:59.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:37:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:37:59.419+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:59.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:59.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:59.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:37:59.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:37:59.374+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:37:59.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:59.237+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:59.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:59.237+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:59.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:59.237+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:37:59.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:37:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:37:59.238+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:00.455 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:00.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:00.456 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:00.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:00.456 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:00.412+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:00.456 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:00.412+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:00.456 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:00.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:00.456 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:00.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:00.456 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:00.280+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:00.457 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:00.281+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:00.454+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:00.454+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:00.454+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:00.455+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:01.442+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:01.442+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:01.442+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:01.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:01.443+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:01.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:01.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:01.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:01.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:01.473+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:01.236+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:01.236+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:01.236+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:01.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:01.237+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:02.471 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:02.440+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:02.472 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:02.440+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:02.472 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:02.440+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:02.472 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:02.440+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:02.472 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:02.215+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:02.472 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:02.215+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:02.473 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:02.215+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:02.473 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:02.216+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:02.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:02.470+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:02.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:02.470+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:02.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:02.470+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:02.471+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:03.403 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:03.297+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18860 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:03.403 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:03.203+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:03.403 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:03.203+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:03.403 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:03.203+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:03.404 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:03.203+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:03.461+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:03.461+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:03.461+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:03.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:03.462+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:03.401+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:03.401+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:03.401+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:03.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:03.402+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:04.430 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:04.355+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:04.431 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:04.355+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:04.431 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:04.355+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:04.431 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:04.356+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:04.431 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:04.171+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:04.431 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:04.171+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:04.431 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:04.171+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:04.432 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:04.172+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:04.429+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:04.429+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:04.429+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:04.430+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:05.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:05.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:05.391+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:05.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:05.392+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:05.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:05.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:05.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:05.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:05.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:05.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:05.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:05.383+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:05.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:05.174+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:05.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:05.174+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:05.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:05.174+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:05.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:05.174+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:06.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:06.135+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:06.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:06.135+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:06.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:06.135+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:06.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:06.136+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:06.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:06.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:06.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:06.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:06.430+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:06.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:06.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:06.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:06.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:06.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:06.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:06.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:06.354+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:07.410 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:07.144+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:07.637 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:07.144+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:07.637 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:07.144+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:07.637 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:07.145+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:07.638 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:07.346+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:07.638 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:07.346+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:07.638 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:07.346+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:07.638 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:07.347+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:07.408+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:07.408+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:07.408+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:07.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:07.409+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:08.422 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:08.299+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 18947 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:08.423 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:08.324+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:08.423 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:08.324+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:08.423 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:08.324+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:08.423 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:08.325+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:08.424 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:08.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:08.424 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:08.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:08.424 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:08.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:08.424 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:08.111+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:08.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:08.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:08.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:08.440+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:09.462 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:09.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:09.462 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:09.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:09.462 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:09.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:09.462 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:09.313+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:09.462 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:09.111+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:09.462 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:09.111+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:09.463 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:09.111+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:09.463 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:09.112+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:09.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:09.460+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:09.460+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:09.460+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:09.461+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:10.510 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:10.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:10.511 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:10.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:10.511 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:10.348+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:10.511 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:10.349+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:10.511 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:10.109+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:10.511 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:10.109+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:10.512 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:10.109+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:10.512 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:10.109+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:10.509+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:10.509+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:10.509+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:10.510+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:11.322+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:11.322+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:11.322+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:11.323+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:11.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:11.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:11.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:11.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:11.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:11.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:11.102+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:11.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:11.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:11.507+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:11.508+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:12.490 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:12.322+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:12.490 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:12.322+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:12.490 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:12.322+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:12.491 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:12.322+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:12.491 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:12.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:12.491 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:12.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:12.491 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:12.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:12.492 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:12.128+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:12.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:12.489+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:12.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:12.489+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:12.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:12.489+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:12.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:12.490+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:13.366 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:13.301+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 19034 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:13.367 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:13.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:13.367 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:13.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:13.368 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:13.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:13.368 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:13.111+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:13.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:13.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:13.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:13.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:13.366+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:13.500+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:13.500+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:13.500+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:13.501+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:14.544 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:14.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:14.545 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:14.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:14.545 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:14.327+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:14.545 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:14.329+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:14.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:14.155+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:14.546 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:14.155+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:14.546 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:14.155+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:14.546 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:14.156+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:14.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:14.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:14.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:14.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:14.544+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:15.582 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:15.361+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:15.582 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:15.361+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:15.582 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:15.361+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:15.582 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:15.362+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:15.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:15.165+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:15.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:15.165+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:15.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:15.165+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:15.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:15.165+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:15.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:15.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:15.581+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:15.581+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:16.544 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:16.412+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:16.544 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:16.412+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:16.545 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:16.412+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:16.545 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:16.413+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:16.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:16.208+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:16.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:16.208+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:16.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:16.208+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:16.545 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:16.209+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:16.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:16.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:16.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:16.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:16.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:16.543+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:16.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:16.544+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:17.526 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:17.377+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:17.526 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:17.377+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:17.526 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:17.377+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:17.526 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:17.378+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:17.526 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:17.217+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:17.527 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:17.217+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:17.527 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:17.217+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:17.527 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:17.217+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:17.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:17.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:17.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:17.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:17.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:17.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:17.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:17.525+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:18.552 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:18.303+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 19121 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:18.553 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:18.385+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:18.553 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:18.385+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:18.553 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:18.385+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:18.553 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:18.386+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:18.553 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:18.209+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:18.553 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:18.209+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:18.554 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:18.209+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:18.554 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:18.209+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:18.551+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:18.551+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:18.551+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:18.552+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:19.535 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:19.416+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:19.535 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:19.416+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:19.535 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:19.416+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:19.535 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:19.417+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:19.536 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:19.248+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:19.536 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:19.248+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:19.536 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:19.248+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:19.536 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:19.249+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:19.534+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:19.534+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:19.534+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:19.535+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:20.533 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:20.396+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:20.533 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:20.396+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:20.533 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:20.396+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:20.533 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:20.397+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:20.534 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:20.204+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:20.534 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:20.204+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:20.534 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:20.204+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:20.534 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:20.204+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:20.532+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:20.532+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:20.532+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:20.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:20.533+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:21.494 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:21.406+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:21.494 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:21.406+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:21.494 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:21.406+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:21.494 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:21.407+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:21.495 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:21.207+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:21.495 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:21.207+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:21.495 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:21.207+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:21.495 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:21.208+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:21.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:21.492+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:21.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:21.492+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:21.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:21.492+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:21.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:21.493+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:22.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:22.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:22.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:22.466+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:22.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:22.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:22.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:22.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:22.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:22.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:22.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:22.457+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:22.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:22.254+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:22.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:22.254+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:22.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:22.254+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:22.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:22.255+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:23.603 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:23.305+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 19212 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:23.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:23.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:23.603 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:23.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:23.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:23.459+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:23.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:23.460+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:23.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:23.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:23.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:23.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:23.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:23.503+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:23.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:23.503+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:23.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:23.276+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:23.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:23.276+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:23.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:23.276+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:23.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:23.277+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:24.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:24.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:24.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:24.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:24.476+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:24.275+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:24.275+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:24.275+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:24.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:24.276+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:24.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:24.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:24.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:24.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:24.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:24.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:24.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:24.433+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:25.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:25.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:25.456+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:25.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:25.457+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:25.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:25.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:25.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:25.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:25.446+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:25.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:25.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:25.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:25.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:25.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:25.306+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:25.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:25.307+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:26.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:26.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:26.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:26.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:26.398+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:26.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:26.464+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:26.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:26.464+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:26.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:26.464+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:26.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:26.464+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:26.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:26.335+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:26.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:26.335+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:26.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:26.335+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:26.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:26.336+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:27.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:27.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:27.445+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:27.446+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:27.313+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:27.313+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:27.313+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:27.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:27.314+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:27.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:27.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:27.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:27.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:27.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:27.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:27.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:27.431+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:28.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:28.306+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 19302 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:28.481+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:28.481+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:28.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:28.481+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:28.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:28.481+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:28.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:28.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:28.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:28.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:28.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:28.351+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:28.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:28.351+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:28.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:28.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:28.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:28.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:28.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:28.490+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:28.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:28.491+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:29.338+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:29.338+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:29.338+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:29.339+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:29.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:29.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:29.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:29.501+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:29.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:29.484+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:29.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:29.484+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:29.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:29.484+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:29.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:29.484+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:30.457 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:30.325+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:30.458 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:30.325+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:30.458 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:30.325+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:30.458 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:30.326+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:30.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:30.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:30.456+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:30.456+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:30.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:30.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:30.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:30.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:30.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:30.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:30.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:30.476+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:31.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:31.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:31.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:31.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:31.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:31.498+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:31.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:31.499+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:31.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:31.462+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:31.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:31.462+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:31.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:31.462+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:31.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:31.463+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:31.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:31.289+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:31.608 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:31.289+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:31.608 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:31.289+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:31.608 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:31.290+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:32.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:32.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:32.473+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:32.474+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:32.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:32.486+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:32.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:32.487+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:32.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:32.487+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:32.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:32.487+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:32.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:32.282+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:32.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:32.282+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:32.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:32.282+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:32.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:32.283+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:33.433 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:33.309+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 19394 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:33.434 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:33.311+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:33.434 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:33.311+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:33.434 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:33.311+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:33.434 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:33.312+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:33.724 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:33.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:33.724 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:33.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:33.724 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:33.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:33.724 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:33.514+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:33.725 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:33.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:33.725 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:33.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:33.725 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:33.508+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:33.725 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:33.508+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:34.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:34.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:34.468+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:34.469+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:34.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:34.479+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:34.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:34.479+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:34.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:34.479+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:34.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:34.480+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:34.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:34.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:34.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:34.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:34.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:34.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:34.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:34.332+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:35.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:35.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:35.441+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:35.441+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:35.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:35.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:35.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:35.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:35.469+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:35.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:35.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:35.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:35.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:35.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:35.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:35.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:35.381+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:36.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:36.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:36.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:36.428+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:36.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:36.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:36.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:36.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:36.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:36.483+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:36.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:36.484+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:36.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:36.372+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:36.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:36.372+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:36.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:36.372+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:36.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:36.372+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:37.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:37.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:37.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:37.473+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:37.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:37.421+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:37.421+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:37.421+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:37.421+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:37.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:37.505+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:37.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:37.505+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:37.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:37.505+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:37.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:37.506+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:38.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:38.311+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 19484 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:38.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:38.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:38.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:38.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:38.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:38.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:38.524+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:38.504+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:38.504+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:38.504+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:38.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:38.505+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:38.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:38.392+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:38.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:38.392+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:38.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:38.392+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:38.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:38.393+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:39.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:39.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:39.354+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:39.355+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:39.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:39.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:39.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:39.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:39.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:39.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:39.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:39.483+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:39.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:39.512+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:39.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:39.512+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:39.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:39.512+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:39.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:39.512+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:40.437 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:40.317+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:40.437 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:40.317+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:40.437 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:40.317+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:40.438 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:40.318+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:40.436+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:40.436+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:40.436+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:40.437+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:40.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:40.541+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:40.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:40.541+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:40.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:40.541+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:40.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:40.542+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:41.337+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:41.337+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:41.337+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:41.337+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:41.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:41.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:41.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:41.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:41.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:41.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:41.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:41.448+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:41.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:41.519+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:41.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:41.519+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:41.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:41.519+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:41.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:41.520+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:42.454+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:42.454+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:42.454+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:42.455+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:42.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:42.568+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:42.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:42.568+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:42.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:42.568+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:42.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:42.569+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:42.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:42.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:42.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:42.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:42.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:42.361+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:42.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:42.362+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:43.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:43.313+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 19575 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:43.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:43.536+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:43.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:43.536+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:43.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:43.536+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:43.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:43.536+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:43.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:43.387+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:43.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:43.387+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:43.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:43.387+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:43.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:43.389+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:43.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:43.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:43.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:43.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:43.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:43.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:43.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:43.462+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:44.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:44.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:44.438+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:44.439+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:44.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:44.555+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:44.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:44.555+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:44.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:44.555+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:44.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:44.555+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:44.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:44.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:44.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:44.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:44.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:44.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:44.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:44.399+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:45.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:45.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:45.460+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:45.461+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:45.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:45.591+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:45.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:45.591+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:45.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:45.591+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:45.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:45.591+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:45.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:45.369+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:45.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:45.369+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:45.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:45.369+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:45.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:45.370+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:46.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:46.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:46.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:46.473+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:46.583+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:46.583+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:46.583+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:46.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:46.584+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:46.350+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:46.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:46.350+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:46.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:46.350+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:46.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:46.351+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:47.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:47.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:47.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:47.487+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:47.593+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:47.593+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:47.593+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:47.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:47.594+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:47.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:47.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:47.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:47.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:47.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:47.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:47.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:47.357+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:48.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:48.315+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 19664 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:48.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:48.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:48.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:48.483+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:48.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:48.484+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:48.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:48.388+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:48.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:48.388+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:48.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:48.388+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:48.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:48.388+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:48.643+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:48.643+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:48.643+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:48.643+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:49.683 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:49.506+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:49.683 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:49.506+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:49.683 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:49.506+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:49.683 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:49.507+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:49.684 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:49.680+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:49.684 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:49.680+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:49.684 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:49.680+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:49.684 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:49.681+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:49.685 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:49.431+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:49.685 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:49.431+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:49.685 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:49.431+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:49.685 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:49.432+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:50.663+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:50.663+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:50.663+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:50.663+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:50.471+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:50.471+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:50.471+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:50.472+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:50.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:50.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:50.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:50.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:50.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:50.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:50.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:50.531+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:51.708 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:51.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:51.708 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:51.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:51.708 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:51.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:51.708 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:51.545+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:51.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:51.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:51.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:51.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:51.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:51.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:51.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:51.435+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:51.709 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:51.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:51.710 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:51.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:51.710 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:51.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:51.710 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:51.645+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:52.666 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:52.514+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:52.666 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:52.514+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:52.666 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:52.514+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:52.667 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:52.515+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:52.667 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:52.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:52.667 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:52.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:52.667 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:52.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:52.667 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:52.408+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:52.665+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:52.665+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:52.665+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:52.666+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:53.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:53.317+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 19756 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:53.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:53.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:53.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:53.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:53.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:53.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:53.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:53.517+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:53.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:53.432+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:53.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:53.432+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:53.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:53.432+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:53.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:53.432+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:54.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:53.663+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:54.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:53.663+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:54.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:53.663+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:54.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:53.664+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:54.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:54.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:54.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:54.514+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:54.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:54.649+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:54.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:54.649+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:54.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:54.649+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:54.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:54.649+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:54.424+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:54.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:54.424+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:54.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:54.424+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:54.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:54.425+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:55.559 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:55.426+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:55.559 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:55.426+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:55.560 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:55.426+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:55.560 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:55.427+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:55.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:55.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:55.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:55.559+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:55.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:55.635+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:55.635+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:55.635+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:55.636+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:56.537+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:56.537+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:56.537+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:56.538+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:56.651+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:56.651+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:56.651+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:56.652+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:56.473+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:56.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:56.473+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:56.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:56.473+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:56.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:56.474+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:57.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:57.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:57.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:57.582+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:57.509+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:57.509+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:57.509+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:57.510+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:57.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:57.658+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:57.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:57.658+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:57.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:57.658+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:57.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:57.659+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:58.603 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:38:58.319+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 19846 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:38:58.603 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:58.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:58.603 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:58.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:58.603 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:58.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:58.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:58.505+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:58.698+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:58.698+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:58.698+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:58.699+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:58.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:58.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:58.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:58.602+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:59.557+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:59.557+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:59.557+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:38:59.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:38:59.557+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:59.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:59.669+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:59.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:59.669+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:59.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:59.669+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:38:59.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:38:59.670+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:38:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:59.547+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:59.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:59.547+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:59.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:59.547+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:38:59.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:38:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:38:59.548+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:00.707+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:00.707+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:00.707+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:00.708+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:00.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:00.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:00.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:00.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:00.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:00.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:00.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:00.599+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:00.557+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:00.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:00.557+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:00.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:00.557+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:00.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:00.558+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:01.632 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:01.516+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:01.633 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:01.516+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:01.633 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:01.516+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:01.633 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:01.516+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:02.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:01.631+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:02.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:01.631+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:02.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:01.631+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:02.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:01.632+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:02.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:01.712+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:02.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:01.712+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:02.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:01.712+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:02.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:01.713+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:02.748 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:02.645+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:02.748 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:02.645+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:02.748 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:02.645+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:02.748 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:02.646+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:02.749 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:02.485+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:02.749 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:02.485+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:02.749 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:02.485+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:02.749 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:02.485+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:02.747+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:02.747+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:02.747+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:02.747+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:03.442 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:03.321+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 19938 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:03.722 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:03.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:03.722 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:03.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:03.722 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:03.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:03.722 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:03.690+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:03.722 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:03.533+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:03.723 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:03.533+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:03.723 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:03.533+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:03.723 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:03.534+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:04.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:03.720+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:03.720+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:03.720+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:03.721+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:04.686+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:04.686+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:04.686+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:04.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:04.687+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:04.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:04.684+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:04.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:04.684+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:04.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:04.684+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:04.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:04.686+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:04.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:04.551+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:04.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:04.551+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:04.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:04.551+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:04.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:04.552+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:05.672+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:05.672+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:05.672+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:05.673+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:05.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:05.695+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:05.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:05.695+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:05.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:05.695+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:05.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:05.695+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:05.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:05.517+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:05.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:05.517+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:05.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:05.518+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:05.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:05.518+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:06.711+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:06.711+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:06.711+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:06.712+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:06.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:06.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:06.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:06.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:06.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:06.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:06.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:06.671+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:06.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:06.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:06.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:06.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:06.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:06.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:06.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:06.559+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:07.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:07.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:07.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:07.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:07.631+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:07.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:07.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:07.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:07.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:07.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:07.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:07.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:07.722+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:07.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:07.593+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:07.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:07.593+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:07.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:07.593+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:07.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:07.594+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:08.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:08.324+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20027 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:08.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:08.566+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:08.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:08.566+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:08.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:08.566+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:08.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:08.567+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:08.679+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:08.679+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:08.679+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:08.680+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:09.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:08.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:09.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:08.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:09.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:08.656+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:09.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:08.657+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:09.703+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:09.703+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:09.703+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:09.704+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:09.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:09.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:09.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:09.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:09.721+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:09.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:09.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:09.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:09.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:09.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:09.576+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:09.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:09.576+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:10.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:10.540+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:10.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:10.540+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:10.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:10.540+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:10.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:10.541+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:10.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:10.688+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:10.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:10.688+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:10.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:10.688+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:10.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:10.689+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:10.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:10.687+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:10.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:10.687+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:10.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:10.687+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:10.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:10.688+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:11.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:11.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:11.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:11.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:11.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:11.535+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:11.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:11.536+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:11.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:11.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:11.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:11.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:11.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:11.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:11.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:11.654+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:11.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:11.729+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:11.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:11.729+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:11.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:11.729+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:11.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:11.730+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:12.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:12.487+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:12.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:12.487+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:12.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:12.487+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:12.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:12.488+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:12.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:12.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:12.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:12.619+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:12.713+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:12.713+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:12.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:12.713+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:12.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:12.714+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:13.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:13.325+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20119 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:13.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:13.448+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:13.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:13.448+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:13.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:13.448+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:13.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:13.449+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:14.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:13.744+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:14.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:13.744+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:14.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:13.744+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:14.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:13.745+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:14.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:13.616+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:14.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:13.616+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:14.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:13.616+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:14.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:13.617+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:14.780 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:14.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:14.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:14.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:14.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:14.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:14.782 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:14.612+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:14.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:14.450+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:14.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:14.450+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:14.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:14.450+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:14.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:14.451+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:15.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:14.778+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:14.778+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:14.778+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:14.779+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:15.831 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:15.613+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:15.832 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:15.613+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:15.832 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:15.613+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:15.832 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:15.615+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:15.832 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:15.403+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:15.832 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:15.403+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:15.832 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:15.403+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:15.833 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:15.404+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:15.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:15.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:15.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:15.830+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:16.819 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:16.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:16.819 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:16.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:16.819 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:16.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:16.819 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:16.390+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:16.819 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:16.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:16.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:16.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:16.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:16.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:16.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:16.593+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:16.817+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:16.817+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:16.817+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:16.818+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:17.602+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:17.602+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:17.602+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:17.602+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:17.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:17.413+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:17.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:17.413+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:17.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:17.413+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:17.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:17.414+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:18.329 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:17.856+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:18.329 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:17.856+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:18.329 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:17.856+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:18.330 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:17.857+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:18.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:18.328+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20205 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:18.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:18.436+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:18.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:18.436+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:18.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:18.436+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:18.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:18.436+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:18.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:18.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:18.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:18.619+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:19.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:18.826+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:19.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:18.826+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:19.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:18.826+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:19.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:18.826+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:19.779 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:19.609+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:19.779 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:19.609+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:19.779 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:19.609+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:19.780 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:19.610+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:19.780 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:19.468+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:19.780 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:19.468+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:19.780 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:19.468+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:19.780 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:19.469+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:19.778+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:19.778+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:19.778+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:19.779+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:20.812 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:20.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:20.812 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:20.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:20.812 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:20.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:20.812 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:20.435+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:20.813 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:20.612+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:20.813 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:20.612+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:20.813 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:20.612+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:20.813 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:20.612+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:20.811+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:20.811+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:20.811+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:20.811+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:21.710 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:21.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:21.710 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:21.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:21.710 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:21.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:21.710 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:21.619+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:21.711 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:21.396+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:21.711 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:21.396+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:21.711 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:21.396+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:21.711 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:21.397+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:21.854+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:21.854+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:21.854+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:21.855+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:22.644+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:22.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:22.644+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:22.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:22.644+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:22.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:22.645+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:22.396+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:22.396+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:22.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:22.396+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:22.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:22.397+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:22.825+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:22.825+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:22.825+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:22.826+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:23.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:23.330+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20287 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:23.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:23.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:23.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:23.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:23.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:23.364+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:23.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:23.364+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:23.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:23.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:23.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:23.640+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:24.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:23.844+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:24.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:23.844+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:24.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:23.844+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:24.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:23.844+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:24.668 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:24.395+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:24.668 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:24.395+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:24.668 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:24.395+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:24.668 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:24.396+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:24.867+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:24.867+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:24.867+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:24.867+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:25.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:24.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:25.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:24.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:25.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:24.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:25.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:24.667+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:25.382+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:25.382+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:25.382+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:25.383+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:25.948 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:25.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:25.948 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:25.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:25.948 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:25.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:25.949 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:25.909+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:25.949 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:25.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:25.949 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:25.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:25.949 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:25.694+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:25.949 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:25.694+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:26.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:26.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:26.343+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:26.344+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:26.917 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:26.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:26.917 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:26.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:26.917 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:26.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:26.917 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:26.647+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:27.325 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:26.916+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:27.325 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:26.916+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:27.325 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:26.916+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:27.325 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:26.916+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:27.323+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:27.323+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:27.323+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:27.324+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:27.942 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:27.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:27.943 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:27.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:27.943 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:27.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:27.943 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:27.627+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:28.322 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:27.941+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:28.323 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:27.941+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:28.323 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:27.941+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:28.323 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:27.942+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:28.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:28.332+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20367 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:28.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:28.321+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:28.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:28.321+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:28.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:28.321+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:28.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:28.322+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:28.898+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:28.898+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:28.898+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:28.899+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:29.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:28.660+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:29.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:28.660+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:29.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:28.660+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:29.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:28.661+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:29.292+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:29.292+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:29.292+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:29.293+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:29.864+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:29.864+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:29.864+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:29.864+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:30.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:29.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:30.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:29.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:30.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:29.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:30.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:29.628+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:30.600 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:30.312+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:30.600 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:30.312+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:30.601 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:30.312+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:30.601 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:30.313+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:30.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:30.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:30.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:30.600+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:31.297 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:30.861+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:31.298 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:30.861+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:31.298 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:30.861+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:31.298 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:30.862+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:31.593 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:31.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:31.621 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:31.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:31.621 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:31.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:31.621 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:31.297+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:31.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:31.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:31.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:31.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:31.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:31.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:31.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:31.593+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:31.847+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:31.847+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:31.847+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:31.848+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:32.584 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:32.301+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:32.585 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:32.301+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:32.585 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:32.301+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:32.585 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:32.302+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:32.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:32.583+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:32.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:32.583+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:32.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:32.583+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:32.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:32.584+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:33.265 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:32.883+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:33.266 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:32.883+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:33.266 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:32.883+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:33.266 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:32.884+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:33.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:33.334+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20449 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:33.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:33.264+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:33.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:33.264+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:33.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:33.264+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:33.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:33.265+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:33.929 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:33.633+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:33.929 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:33.633+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:33.929 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:33.633+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:33.930 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:33.634+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:34.255 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:33.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:34.255 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:33.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:34.255 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:33.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:34.255 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:33.929+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:34.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:34.254+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:34.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:34.254+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:34.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:34.254+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:34.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:34.254+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:34.908 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:34.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:34.908 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:34.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:34.909 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:34.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:34.909 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:34.627+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:35.164 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:34.907+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:35.164 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:34.907+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:35.164 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:34.907+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:35.164 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:34.908+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:35.597 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:35.286+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:35.598 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:35.286+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:35.598 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:35.286+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:35.598 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:35.286+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:35.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:35.596+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:35.596+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:35.596+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:35.597+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:36.334 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:35.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:36.335 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:35.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:36.335 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:35.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:36.335 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:35.929+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:36.587 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:36.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:36.587 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:36.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:36.588 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:36.333+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:36.588 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:36.334+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:36.586+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:36.586+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:36.586+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:36.587+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:37.287 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:36.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:37.287 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:36.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:37.287 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:36.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:37.287 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:36.943+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:37.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:37.285+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:37.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:37.285+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:37.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:37.285+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:37.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:37.286+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:37.983 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:37.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:37.983 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:37.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:37.983 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:37.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:37.983 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:37.612+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:38.291 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:37.982+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:38.292 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:37.982+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:38.292 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:37.982+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:38.292 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:37.983+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:38.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:38.337+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20529 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:38.290+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:38.290+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:38.290+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:38.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:38.291+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:39.019 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:38.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:39.020 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:38.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:39.020 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:38.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:39.020 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:38.659+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:39.279 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:39.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:39.279 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:39.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:39.280 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:39.018+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:39.280 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:39.019+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:39.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:39.278+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:39.658 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:39.278+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:39.659 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:39.278+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:39.659 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:39.279+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:39.977 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:39.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:39.977 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:39.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:39.977 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:39.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:39.978 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:39.655+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:40.284 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:39.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:40.284 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:39.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:40.284 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:39.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:40.284 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:39.977+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:40.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:40.283+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:40.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:40.283+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:40.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:40.283+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:40.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:40.283+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:40.979 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:40.684+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:40.979 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:40.684+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:40.979 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:40.684+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:40.980 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:40.685+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:41.237 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:40.978+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:41.238 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:40.978+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:41.238 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:40.978+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:41.238 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:40.979+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:41.236+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:41.236+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:41.236+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:41.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:41.237+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:41.983 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:41.636+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:41.983 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:41.636+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:41.984 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:41.636+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:41.984 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:41.637+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:41.982+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:41.982+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:41.982+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:41.983+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:42.207+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:42.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:42.207+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:42.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:42.207+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:42.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:42.208+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:42.950 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:42.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:42.951 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:42.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:42.951 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:42.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:42.951 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:42.615+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:43.235 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:42.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:43.235 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:42.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:43.236 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:42.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:43.236 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:42.950+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:43.593 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:43.339+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20617 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:43.594 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:43.234+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:43.594 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:43.234+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:43.594 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:43.234+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:43.594 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:43.235+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:43.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:43.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:43.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:43.593+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:44.275 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:43.900+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:44.275 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:43.900+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:44.275 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:43.900+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:44.275 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:43.901+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:44.556 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:44.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:44.556 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:44.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:44.556 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:44.271+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:44.556 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:44.272+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:44.555+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:44.555+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:44.555+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:44.556+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:45.300 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:44.872+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:45.301 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:44.872+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:45.301 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:44.872+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:45.301 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:44.872+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:45.593 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:45.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:45.593 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:45.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:45.593 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:45.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:45.593 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:45.300+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:45.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:45.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:45.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:45.592+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:45.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:45.593+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:46.308 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:45.892+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:46.309 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:45.892+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:46.309 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:45.892+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:46.309 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:45.893+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:46.307+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:46.307+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:46.307+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:46.308+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:46.887 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:46.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:46.887 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:46.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:46.887 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:46.634+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:46.887 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:46.635+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:47.329 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:46.886+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:47.329 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:46.886+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:47.329 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:46.886+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:47.329 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:46.886+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:47.328+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:47.328+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:47.328+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:47.329+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:47.617+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:47.617+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:47.617+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:47.618+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:47.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:48.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:47.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:48.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:47.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:48.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:47.851+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:48.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:48.341+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20707 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:48.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:48.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:48.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:48.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:48.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:48.366+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:48.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:48.367+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:48.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:48.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:48.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:48.603+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:48.604+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:48.898+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:48.898+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:48.898+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:48.898+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:49.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:49.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:49.605+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:49.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:49.605+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:49.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:49.397+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:49.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:49.397+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:49.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:49.397+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:49.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:49.398+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:50.181 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:49.859+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:50.181 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:49.859+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:50.181 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:49.859+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:50.181 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:49.859+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:50.524 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:50.410+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:50.524 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:50.410+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:50.524 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:50.410+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:50.524 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:50.411+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:50.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:50.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:50.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:50.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:50.559+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:51.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:50.898+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:50.898+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:50.898+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:50.899+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:51.711 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:51.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:51.712 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:51.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:51.712 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:51.546+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:51.712 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:51.547+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:51.712 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:51.453+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:51.712 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:51.453+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:51.712 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:51.453+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:51.713 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:51.454+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:52.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:51.916+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:52.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:51.916+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:52.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:51.916+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:52.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:51.917+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:52.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:52.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:52.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:52.546+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:52.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:52.441+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:52.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:52.441+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:52.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:52.441+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:52.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:52.442+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:52.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:52.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:52.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:52.929+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:53.596 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:53.343+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20799 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:53.596 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:53.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:53.597 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:53.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:53.597 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:53.513+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:53.597 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:53.513+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:53.597 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:53.453+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:53.597 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:53.453+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:53.597 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:53.453+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:53.598 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:53.454+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:53.909+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:53.909+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:53.909+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:53.909+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:54.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:54.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:54.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:54.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:54.547+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:54.477+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:54.477+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:54.477+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:54.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:54.478+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:54.909+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:54.909+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:54.909+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:54.910+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:55.594+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:55.594+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:55.594+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:55.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:55.595+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:55.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:55.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:55.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:55.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:55.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:55.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:55.430+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:55.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:55.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:55.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:55.909+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:56.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:56.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:56.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:56.615+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:56.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:56.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:56.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:56.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:56.431+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:56.932+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:56.932+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:56.932+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:56.932+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:57.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:57.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:57.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:57.611+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:57.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:57.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:57.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:57.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:57.431+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:58.347 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:57.965+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:58.347 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:57.965+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:58.347 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:57.965+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:58.347 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:57.965+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:58.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:39:58.344+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20889 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:39:58.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:58.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:58.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:58.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:58.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:58.416+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:58.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:58.416+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:58.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:58.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:58.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:58.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:58.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:58.582+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:58.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:58.582+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:58.983+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:58.983+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:58.983+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:39:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:58.984+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:59.630 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:59.367+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:59.630 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:59.367+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:59.630 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:59.367+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:39:59.630 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:39:59.367+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:39:59.958 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:59.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:59.958 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:59.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:59.958 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:59.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:39:59.959 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:39:59.629+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:00.326 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:59.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:00.326 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:59.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:00.326 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:59.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:00.326 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:39:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:39:59.957+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:00.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:00.323+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:00.323+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:00.323+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:00.324+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:00.936 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:00.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:00.936 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:00.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:00.936 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:00.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:00.936 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:00.619+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:01.334 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:00.934+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:01.335 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:00.934+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:01.335 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:00.934+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:01.335 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:00.935+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:01.595 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:01.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:01.596 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:01.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:01.596 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:01.332+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:01.596 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:01.333+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:01.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:01.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:01.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:01.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:01.594+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:01.952+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:01.952+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:01.952+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:01.952+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:02.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:02.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:02.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:02.593+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:02.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:02.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:02.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:02.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:02.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:02.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:02.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:02.356+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:02.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:02.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:02.928+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:02.928+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:03.359 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:03.347+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 20979 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:03.360 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:03.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:03.360 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:03.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:03.360 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:03.345+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:03.360 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:03.345+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:03.610 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:03.585+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:03.610 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:03.585+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:03.610 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:03.585+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:03.610 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:03.586+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:04.335 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:03.962+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:04.336 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:03.962+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:04.336 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:03.962+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:04.336 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:03.962+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:04.334+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:04.334+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:04.334+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:04.335+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:04.977 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:04.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:04.977 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:04.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:04.977 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:04.611+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:04.977 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:04.613+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:05.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:04.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:05.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:04.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:05.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:04.976+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:05.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:04.976+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:05.630 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:05.360+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:05.631 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:05.360+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:05.631 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:05.360+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:05.631 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:05.361+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:05.884 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:05.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:05.884 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:05.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:05.884 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:05.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:05.885 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:05.631+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:06.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:05.952+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:06.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:05.952+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:06.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:05.952+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:06.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:05.953+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:06.657 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:06.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:06.657 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:06.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:06.657 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:06.383+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:06.657 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:06.384+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:06.995 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:06.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:06.995 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:06.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:06.995 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:06.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:06.995 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:06.656+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:07.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:06.993+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:07.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:06.993+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:07.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:06.993+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:07.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:06.995+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:07.680 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:07.425+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:07.680 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:07.425+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:07.680 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:07.425+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:07.681 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:07.426+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:08.006 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:07.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:08.006 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:07.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:08.006 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:07.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:08.006 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:07.680+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:08.351 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:08.004+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:08.351 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:08.004+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:08.351 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:08.004+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:08.351 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:08.005+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:08.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:08.350+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 21069 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:08.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:08.401+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:08.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:08.401+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:08.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:08.401+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:08.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:08.401+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:08.998 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:08.725+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:08.999 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:08.725+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:08.999 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:08.725+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:08.999 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:08.726+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:08.997+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:08.997+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:08.997+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:08.998+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:09.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:09.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:09.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:09.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:09.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:09.407+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:09.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:09.407+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:10.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:09.708+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:10.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:09.708+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:10.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:09.708+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:10.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:09.708+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:10.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:10.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:10.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:10.035+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:10.744 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:10.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:10.744 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:10.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:10.744 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:10.415+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:10.744 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:10.416+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:11.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:10.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:11.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:10.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:11.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:10.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:11.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:10.743+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:11.039+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:11.039+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:11.039+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:11.041+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:11.712 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:11.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:11.712 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:11.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:11.713 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:11.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:11.713 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:11.390+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:12.028 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:11.734+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:12.028 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:11.734+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:12.028 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:11.734+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:12.028 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:11.734+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:12.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:12.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:12.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:12.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:12.083+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:12.689 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:12.362+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:12.690 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:12.362+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:12.690 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:12.362+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:12.690 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:12.363+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:13.042 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:12.688+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:13.042 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:12.688+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:13.042 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:12.688+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:13.042 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:12.689+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:13.353 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:13.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:13.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:13.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:13.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:13.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:13.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:13.041+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:13.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:13.352+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 21162 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:13.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:13.402+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:13.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:13.402+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:13.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:13.402+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:13.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:13.402+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:14.058 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:13.641+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:14.058 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:13.641+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:14.058 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:13.641+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:14.058 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:13.642+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:14.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:14.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:14.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:14.058+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:14.669 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:14.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:14.669 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:14.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:14.670 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:14.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:14.670 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:14.358+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:15.083 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:14.668+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:15.083 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:14.668+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:15.083 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:14.668+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:15.083 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:14.669+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:15.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:15.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:15.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:15.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:15.083+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:15.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:15.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:15.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:15.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:15.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:15.322+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:15.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:15.323+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:16.078 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:15.707+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:16.078 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:15.707+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:16.078 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:15.707+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:16.078 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:15.707+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:16.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:16.077+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:16.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:16.077+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:16.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:16.077+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:16.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:16.078+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:16.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:16.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:16.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:16.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:16.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:16.339+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:16.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:16.339+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:17.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:16.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:17.099 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:16.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:17.099 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:16.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:17.099 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:16.666+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:17.353 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:17.097+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:17.353 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:17.097+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:17.353 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:17.097+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:17.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:17.098+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:17.352+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:17.352+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:17.352+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:17.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:17.353+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:18.057 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:17.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:18.058 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:17.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:18.058 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:17.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:18.058 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:17.663+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:18.056+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:18.056+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:18.056+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:18.057+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:18.678 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:18.354+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 21251 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:18.678 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:18.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:18.678 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:18.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:18.679 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:18.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:18.679 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:18.376+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:19.013 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:18.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:19.014 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:18.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:19.014 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:18.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:19.014 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:18.678+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:19.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:19.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:19.013+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:19.014+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:19.690 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:19.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:19.691 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:19.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:19.691 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:19.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:19.691 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:19.376+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:20.042 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:19.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:20.042 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:19.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:20.042 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:19.689+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:20.042 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:19.690+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:20.041+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:20.041+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:20.041+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:20.041+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:20.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:20.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:20.640+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:20.642+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:20.412+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:20.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:20.412+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:20.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:20.412+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:20.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:20.413+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:21.043+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:21.043+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:21.043+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:21.044+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:21.666 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:21.382+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:21.666 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:21.382+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:21.667 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:21.382+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:21.667 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:21.382+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:22.060 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:21.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:22.061 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:21.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:22.061 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:21.665+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:22.061 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:21.666+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:22.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:22.059+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:22.059+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:22.059+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:22.060+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:22.665 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:22.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:22.665 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:22.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:22.665 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:22.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:22.665 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:22.358+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:23.080 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:22.664+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:23.081 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:22.664+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:23.081 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:22.664+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:23.081 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:22.665+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:23.079+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:23.079+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:23.079+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:23.080+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:23.611 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:23.356+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 21341 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:23.611 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:23.406+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:23.611 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:23.406+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:23.611 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:23.406+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:23.612 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:23.407+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:24.065 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:23.648+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:24.065 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:23.648+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:24.065 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:23.648+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:24.065 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:23.649+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:24.317 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:24.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:24.317 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:24.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:24.317 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:24.064+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:24.318 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:24.065+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:24.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:24.413+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:24.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:24.413+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:24.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:24.413+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:24.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:24.414+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:25.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:24.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:25.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:24.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:25.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:24.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:25.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:24.637+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:25.097+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:25.097+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:25.097+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:25.098+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:25.613+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:25.613+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:25.613+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:25.614+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:25.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:25.421+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:25.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:25.421+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:25.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:25.421+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:25.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:25.422+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:26.080+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:26.080+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:26.080+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:26.081+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:26.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:26.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:26.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:26.574+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:26.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:26.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:26.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:26.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:26.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:26.381+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:26.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:26.382+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:27.071+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:27.071+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:27.071+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:27.071+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:27.567+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:27.567+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:27.567+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:27.568+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:27.402+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:27.402+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:27.402+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:27.402+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:28.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:28.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:28.074+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:28.075+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:28.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:28.358+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 21430 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:28.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:28.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:28.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:28.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:28.601+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:28.384+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:28.384+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:28.384+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:28.384+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:29.342 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:29.077+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:29.343 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:29.077+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:29.343 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:29.077+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:29.343 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:29.078+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:29.341+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:29.341+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:29.341+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:29.342+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:30.040 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:29.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:30.040 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:29.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:30.040 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:29.628+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:30.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:29.629+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:30.039+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:30.039+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:30.039+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:30.040+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:30.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:30.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:30.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:30.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:30.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:30.389+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:30.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:30.389+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:30.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:30.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:30.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:30.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:30.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:30.627+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:30.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:30.627+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:31.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:31.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:31.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:31.041+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:31.714 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:31.607+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:31.714 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:31.607+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:31.714 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:31.607+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:31.715 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:31.607+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:31.715 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:31.372+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:31.715 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:31.372+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:31.715 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:31.372+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:31.715 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:31.373+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:32.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:32.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:32.037+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:32.037+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:32.624+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:32.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:32.624+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:32.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:32.624+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:32.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:32.625+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:32.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:32.385+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:32.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:32.385+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:32.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:32.386+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:32.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:32.387+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:33.034+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:33.034+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:33.034+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:33.035+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:33.622 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:33.361+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 21522 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:33.622 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:33.401+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:33.622 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:33.401+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:33.622 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:33.401+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:33.622 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:33.401+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:34.062 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:33.621+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:34.063 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:33.621+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:34.063 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:33.621+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:34.063 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:33.622+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:34.061+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:34.061+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:34.061+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:34.063+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:34.660 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:34.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:34.661 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:34.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:34.661 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:34.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:34.661 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:34.357+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:35.084 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:34.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:35.084 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:34.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:35.084 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:34.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:35.084 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:34.660+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:35.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:35.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:35.343 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:35.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:35.343 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:35.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:35.343 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:35.083+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:35.387+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:35.387+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:35.387+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:35.387+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:36.083 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:35.675+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:36.084 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:35.675+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:36.084 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:35.675+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:36.084 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:35.676+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:36.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:36.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:36.082+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:36.083+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:36.713 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:36.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:36.713 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:36.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:36.713 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:36.419+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:36.713 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:36.419+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:37.045 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:36.712+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:37.045 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:36.712+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:37.045 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:36.712+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:37.046 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:36.713+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:37.044+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:37.044+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:37.044+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:37.045+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:37.687+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:37.687+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:37.687+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:37.688+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:37.442+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:37.442+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:37.442+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:37.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:37.442+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:38.066+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:38.066+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:38.066+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:38.067+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:38.688 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:38.363+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 21610 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:38.689 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:38.427+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:38.689 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:38.427+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:38.689 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:38.427+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:38.689 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:38.428+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:39.045 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:38.687+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:39.046 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:38.687+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:39.046 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:38.687+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:39.046 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:38.688+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:39.044+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:39.044+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:39.044+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:39.045+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:39.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:39.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:39.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:39.647+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:39.432+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:39.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:39.432+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:39.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:39.432+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:39.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:39.432+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:40.085+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:40.085+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:40.085+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:40.086+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:40.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:40.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:40.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:40.678+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:40.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:40.445+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:40.445+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:40.445+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:40.445+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:41.118+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:41.118+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:41.118+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:41.119+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:41.682+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:41.682+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:41.682+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:41.682+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:41.488+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:41.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:41.488+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:41.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:41.488+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:41.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:41.489+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:42.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:42.164+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:42.529 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:42.164+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:42.529 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:42.164+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:42.529 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:42.165+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:42.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:42.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:42.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:42.648+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:42.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:42.527+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:42.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:42.527+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:42.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:42.527+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:42.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:42.528+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:43.490 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:43.365+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 21702 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:43.490 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:43.201+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:43.491 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:43.201+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:43.491 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:43.201+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:43.491 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:43.202+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:43.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:43.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:43.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:43.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:43.601+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:43.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:43.491+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:43.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:43.491+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:43.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:43.491+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:43.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:43.492+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:44.478 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:44.200+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:44.478 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:44.200+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:44.478 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:44.200+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:44.478 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:44.201+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:44.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:44.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:44.806 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:44.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:44.806 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:44.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:44.806 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:44.626+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:44.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:44.476+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:44.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:44.476+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:44.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:44.476+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:44.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:44.476+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:45.486 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:45.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:45.486 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:45.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:45.487 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:45.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:45.487 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:45.220+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:45.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:45.638+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:45.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:45.638+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:45.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:45.638+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:45.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:45.640+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:45.821 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:45.484+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:45.821 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:45.484+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:45.821 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:45.484+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:45.821 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:45.484+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:46.489 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:46.226+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:46.490 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:46.226+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:46.490 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:46.226+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:46.490 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:46.226+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:46.613+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:46.613+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:46.613+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:46.614+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:46.488+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:46.488+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:46.488+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:46.488+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:47.271+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:47.271+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:47.271+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:47.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:47.271+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:47.484+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:47.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:47.484+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:47.484+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:47.485+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:47.615+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:47.615+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:47.615+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:48.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:47.615+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:48.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:48.367+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 21791 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:48.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:48.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:48.293+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:48.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:48.294+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:48.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:48.441+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:48.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:48.441+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:48.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:48.441+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:48.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:48.442+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:48.648+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:48.648+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:48.648+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:48.649+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:49.467+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:49.467+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:49.467+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:49.467+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:49.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:49.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:49.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:49.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:49.306+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:49.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:49.306+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:49.695+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:49.695+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:49.695+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:49.696+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:50.435 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:50.433+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:50.435 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:50.433+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:50.435 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:50.433+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:50.435 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:50.434+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:50.436 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:50.319+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:50.436 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:50.319+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:50.436 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:50.319+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:50.436 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:50.320+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:51.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:50.690+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:50.691+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:50.691+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:50.691+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:51.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:51.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:51.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:51.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:51.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:51.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:51.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:51.458+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:51.274+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:51.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:51.274+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:51.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:51.274+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:51.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:51.274+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:51.736+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:51.736+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:51.736+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:51.737+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:52.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:52.409+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:52.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:52.409+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:52.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:52.409+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:52.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:52.410+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:52.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:52.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:52.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:52.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:52.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:52.291+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:52.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:52.292+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:52.783+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:52.783+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:52.783+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:52.784+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:53.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:53.370+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 21884 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:53.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:53.336+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:53.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:53.336+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:53.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:53.336+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:53.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:53.337+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:53.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:53.368+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:53.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:53.368+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:53.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:53.368+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:53.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:53.369+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:54.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:53.807+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:54.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:53.807+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:54.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:53.807+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:54.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:53.808+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:54.831 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:54.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:54.831 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:54.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:54.831 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:54.374+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:54.831 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:54.374+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:54.831 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:54.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:54.832 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:54.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:54.832 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:54.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:54.832 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:54.381+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:55.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:54.830+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:55.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:54.830+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:55.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:54.830+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:55.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:54.830+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:55.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:55.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:55.365+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:55.365+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:55.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:55.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:55.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:55.376+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:55.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:55.377+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:56.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:55.854+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:56.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:55.854+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:56.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:55.854+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:56.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:55.855+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:56.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:56.400+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:56.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:56.400+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:56.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:56.400+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:56.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:56.401+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:56.825 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:56.374+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:56.825 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:56.374+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:56.825 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:56.374+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:56.825 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:56.375+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:57.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:56.823+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:57.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:56.823+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:57.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:56.823+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:57.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:56.824+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:57.792 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:57.401+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:57.793 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:57.401+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:57.793 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:57.401+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:57.793 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:57.402+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:57.793 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:57.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:57.793 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:57.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:57.793 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:57.398+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:57.793 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:57.399+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:58.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:57.791+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:58.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:57.791+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:58.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:57.791+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:58.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:57.792+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:58.823 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:40:58.372+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 21973 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:40:58.823 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:58.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:58.823 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:58.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:58.823 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:58.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:58.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:58.421+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:58.824 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:58.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:58.824 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:58.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:58.824 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:58.404+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:58.824 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:58.405+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:59.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:58.822+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:59.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:58.822+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:59.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:58.822+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:40:59.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:58.823+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:59.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:59.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:59.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:59.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:59.824 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:59.465+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:40:59.825 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:40:59.466+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:40:59.825 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:59.402+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:59.825 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:59.402+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:59.825 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:59.402+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:40:59.825 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:40:59.403+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:00.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:59.823+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:00.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:59.823+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:00.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:59.823+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:00.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:40:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:40:59.824+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:00.823 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:00.444+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:00.823 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:00.444+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:00.823 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:00.444+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:00.823 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:00.445+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:00.824 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:00.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:00.824 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:00.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:00.824 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:00.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:00.824 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:00.453+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:01.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:00.822+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:01.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:00.822+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:01.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:00.822+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:01.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:00.823+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:01.795 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:01.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:01.795 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:01.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:01.795 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:01.468+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:01.796 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:01.469+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:01.796 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:01.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:01.796 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:01.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:01.796 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:01.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:01.796 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:01.490+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:02.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:01.793+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:02.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:01.793+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:02.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:01.793+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:02.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:01.795+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:02.781 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:02.440+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:02.781 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:02.440+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:02.781 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:02.440+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:02.781 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:02.441+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:02.781 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:02.526+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:02.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:02.527+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:02.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:02.527+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:02.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:02.527+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:03.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:02.779+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:03.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:02.779+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:03.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:02.779+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:03.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:02.780+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:03.782 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:03.374+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 22065 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:03.783 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:03.479+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:03.783 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:03.479+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:03.783 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:03.479+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:03.783 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:03.480+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:03.783 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:03.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:03.783 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:03.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:03.784 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:03.505+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:03.784 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:03.505+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:04.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:03.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:04.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:03.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:04.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:03.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:04.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:03.782+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:04.797 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:04.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:04.797 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:04.527+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:04.797 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:04.528+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:04.797 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:04.529+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:04.798 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:04.554+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:04.798 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:04.554+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:04.798 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:04.554+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:04.798 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:04.555+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:04.795+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:04.795+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:04.795+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:05.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:04.796+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:05.754 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:05.519+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:05.755 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:05.519+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:05.755 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:05.519+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:05.755 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:05.520+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:05.755 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:05.488+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:05.755 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:05.488+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:05.756 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:05.488+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:05.756 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:05.488+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:06.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:05.753+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:06.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:05.753+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:06.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:05.753+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:06.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:05.754+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:06.782 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:06.487+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:06.782 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:06.487+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:06.782 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:06.487+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:06.782 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:06.487+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:06.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:06.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:06.783 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:06.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:06.783 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:06.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:06.783 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:06.545+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:07.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:06.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:07.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:06.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:07.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:06.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:07.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:06.781+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:07.736 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:07.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:07.737 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:07.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:07.737 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:07.439+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:07.737 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:07.439+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:07.737 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:07.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:07.737 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:07.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:07.737 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:07.560+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:07.738 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:07.560+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:08.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:07.735+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:08.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:07.735+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:08.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:07.735+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:08.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:07.736+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:08.697 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:08.377+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 22154 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:08.697 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:08.393+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:08.697 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:08.393+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:08.698 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:08.393+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:08.698 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:08.394+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:08.698 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:08.580+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:08.698 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:08.580+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:08.698 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:08.580+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:08.698 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:08.580+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:09.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:08.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:09.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:08.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:09.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:08.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:09.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:08.697+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:09.344+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:09.344+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:09.344+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:09.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:09.345+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:09.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:09.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:09.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:09.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:09.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:09.558+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:09.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:09.559+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:10.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:09.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:10.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:09.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:10.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:09.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:10.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:09.651+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:10.565 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:10.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:10.565 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:10.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:10.565 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:10.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:10.565 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:10.305+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:10.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:10.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:10.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:10.695+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:10.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:10.563+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:10.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:10.563+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:10.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:10.563+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:10.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:10.565+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:11.595 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:11.299+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:11.595 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:11.299+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:11.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:11.299+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:11.596 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:11.299+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:11.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:11.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:11.693+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:11.693+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:11.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:11.594+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:11.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:11.594+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:11.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:11.594+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:11.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:11.595+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:12.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:12.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:12.327+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:12.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:12.328+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:13.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:12.644+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:13.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:12.644+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:13.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:12.644+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:13.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:12.645+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:13.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:12.625+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:12.625+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:12.625+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:12.625+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:13.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:13.379+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 22247 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:13.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:13.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:13.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:13.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:13.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:13.334+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:13.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:13.335+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:13.674+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:13.674+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:13.674+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:13.675+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:14.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:13.643+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:14.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:13.643+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:14.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:13.643+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:14.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:13.644+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:14.640 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:14.367+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:14.640 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:14.367+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:14.640 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:14.367+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:14.640 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:14.368+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:14.641 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:14.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:14.641 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:14.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:14.641 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:14.595+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:14.641 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:14.596+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:14.638+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:14.638+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:14.638+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:14.639+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:15.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:15.324+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:15.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:15.324+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:15.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:15.324+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:15.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:15.324+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:16.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:15.638+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:16.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:15.638+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:16.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:15.638+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:16.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:15.638+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:16.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:15.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:16.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:15.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:15.662+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:15.663+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:16.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:16.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:16.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:16.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:16.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:16.328+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:16.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:16.328+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:17.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:16.684+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:17.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:16.684+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:17.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:16.684+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:17.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:16.684+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:17.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:16.632+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:16.632+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:16.632+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:16.633+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:17.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:17.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:17.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:17.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:17.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:17.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:17.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:17.306+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:18.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:17.728+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:18.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:17.728+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:18.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:17.728+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:18.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:17.729+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:18.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:17.604+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:18.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:17.604+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:18.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:17.604+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:18.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:17.605+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:18.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:18.382+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 22336 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:18.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:18.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:18.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:18.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:18.331+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:19.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:18.616+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:19.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:18.616+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:19.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:18.616+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:19.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:18.617+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:18.688+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:19.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:18.688+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:19.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:18.688+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:19.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:18.689+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:19.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:19.352+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:19.352+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:19.352+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:19.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:19.353+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:19.618+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:19.618+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:19.618+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:20.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:19.619+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:19.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:19.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:20.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:19.647+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:20.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:19.648+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:20.574 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:20.573+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:20.575 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:20.573+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:20.575 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:20.573+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:20.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:20.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:20.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:20.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:20.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:20.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:20.575 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:20.331+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:20.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:20.574+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:20.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:20.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:20.626+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:20.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:20.626+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:21.526+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:21.526+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:21.526+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:21.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:21.527+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:21.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:21.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:21.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:21.304+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:21.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:21.305+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:21.649+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:21.649+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:21.649+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:22.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:21.649+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:22.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:22.486+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:22.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:22.486+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:22.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:22.486+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:22.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:22.488+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:22.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:22.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:22.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:22.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:22.309+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:22.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:22.309+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:22.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:22.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:22.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:22.651+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:23.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:23.384+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 22428 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:23.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:23.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:23.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:23.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:23.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:23.339+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:23.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:23.339+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:23.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:23.451+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:23.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:23.451+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:23.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:23.451+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:23.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:23.452+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:23.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:23.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:23.606+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:23.606+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:24.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:24.312+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:24.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:24.312+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:24.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:24.312+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:24.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:24.312+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:24.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:24.414+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:24.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:24.414+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:24.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:24.414+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:24.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:24.414+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:24.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:24.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:24.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:24.601+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:25.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:25.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:25.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:25.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:25.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:25.350+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:25.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:25.350+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:25.457+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:25.457+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:25.457+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:25.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:25.458+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:25.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:25.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:25.601+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:25.602+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:26.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:26.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:26.346+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:26.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:26.347+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:26.496+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:26.496+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:26.496+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:26.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:26.497+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:26.638+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:26.638+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:26.638+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:26.639+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:27.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:27.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:27.599+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:27.600+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:27.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:27.472+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:27.472+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:27.472+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:27.473+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:27.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:27.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:27.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:27.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:27.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:27.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:27.857 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:27.358+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:28.604 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:28.386+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 22517 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:28.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:28.468+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:28.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:28.468+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:28.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:28.468+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:28.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:28.469+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:28.563+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:28.563+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:28.563+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:28.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:28.564+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:28.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:28.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:28.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:28.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:28.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:28.340+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:28.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:28.340+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:29.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:29.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:29.330+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:29.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:29.331+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:29.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:29.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:29.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:29.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:29.537+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:29.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:29.517+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:29.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:29.517+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:29.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:29.517+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:29.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:29.518+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:30.550 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:30.302+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:30.550 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:30.302+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:30.550 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:30.303+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:30.550 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:30.304+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:30.563+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:30.563+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:30.563+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:30.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:30.564+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:30.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:30.548+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:30.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:30.548+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:30.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:30.548+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:30.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:30.548+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:31.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:31.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:31.517+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:31.518+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:31.303+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:31.303+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:31.303+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:31.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:31.303+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:31.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:31.548+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:31.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:31.548+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:31.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:31.548+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:31.607 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:31.549+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:32.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:32.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:32.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:32.537+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:32.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:32.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:32.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:32.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:32.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:32.521+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:32.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:32.522+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:32.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:32.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:32.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:32.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:32.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:32.305+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:32.607 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:32.306+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:33.390 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:33.297+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:33.390 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:33.297+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:33.391 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:33.297+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:33.391 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:33.298+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:33.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:33.388+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 22610 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:33.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:33.570+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:33.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:33.570+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:33.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:33.570+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:33.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:33.571+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:33.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:33.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:33.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:33.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:33.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:33.558+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:33.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:33.559+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:34.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:34.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:34.331+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:34.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:34.332+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:34.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:34.531+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:34.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:34.531+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:34.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:34.531+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:34.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:34.531+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:34.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:34.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:34.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:34.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:34.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:34.545+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:34.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:34.546+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:35.545 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:35.292+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:35.546 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:35.292+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:35.546 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:35.292+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:35.546 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:35.293+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:35.545+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:35.545+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:35.545+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:35.546+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:35.587+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:35.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:35.587+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:35.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:35.587+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:35.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:35.588+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:36.509 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:36.257+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:36.509 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:36.257+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:36.509 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:36.257+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:36.509 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:36.258+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:36.508+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:36.508+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:36.508+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:36.509+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:36.591+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:36.591+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:36.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:36.591+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:36.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:36.592+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:37.581 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:37.250+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:37.581 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:37.250+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:37.581 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:37.250+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:37.581 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:37.252+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:37.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:37.487+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:37.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:37.487+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:37.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:37.487+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:37.582 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:37.488+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:37.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:37.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:37.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:37.580+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:37.581+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:38.582 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:38.391+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 22699 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:38.582 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:38.301+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:38.582 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:38.301+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:38.582 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:38.301+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:38.583 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:38.302+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:38.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:38.461+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:38.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:38.461+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:38.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:38.461+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:38.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:38.461+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:38.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:38.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:38.581+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:38.582+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:39.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:39.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:39.602 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:39.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:39.602 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:39.265+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:39.602 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:39.266+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:39.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:39.509+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:39.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:39.509+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:39.602 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:39.509+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:39.603 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:39.510+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:39.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:39.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:39.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:39.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:39.601+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:40.517 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:40.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:40.517 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:40.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:40.517 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:40.219+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:40.518 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:40.220+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:40.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:40.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:40.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:40.575+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:40.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:40.516+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:40.516+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:40.516+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:40.517+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:41.527 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:41.229+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:41.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:41.229+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:41.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:41.229+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:41.528 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:41.229+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:41.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:41.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:41.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:41.585+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:41.526+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:41.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:41.526+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:41.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:41.526+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:41.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:41.527+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:42.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:42.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:42.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:42.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:42.569 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:42.242+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:42.570 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:42.243+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:42.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:42.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:42.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:42.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:42.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:42.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:42.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:42.630+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:42.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:42.568+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:42.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:42.568+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:42.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:42.568+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:42.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:42.569+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:43.520 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:43.394+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 22791 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:43.520 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:43.195+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:43.521 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:43.195+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:43.521 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:43.195+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:43.521 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:43.196+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:43.661+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:43.661+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:43.661+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:43.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:43.662+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:43.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:43.601+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:43.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:43.601+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:43.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:43.601+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:43.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:43.602+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:44.182+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:44.182+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:44.182+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:44.183+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:44.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:44.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:44.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:44.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:44.678+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:44.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:44.578+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:44.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:44.578+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:44.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:44.578+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:44.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:44.579+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:45.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:45.150+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:45.150+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:45.150+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:45.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:45.150+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:45.680+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:45.680+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:45.680+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:45.681+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:45.620+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:45.620+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:46.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:45.620+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:46.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:45.621+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:46.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:46.129+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:46.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:46.129+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:46.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:46.129+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:46.601 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:46.129+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:46.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:46.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:46.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:46.676+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:46.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:46.677+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:46.600+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:46.600+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:46.600+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:46.600+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:47.598 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:47.162+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:47.598 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:47.162+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:47.599 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:47.162+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:47.599 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:47.163+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:47.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:47.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:47.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:47.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:47.697+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:47.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:47.597+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:47.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:47.597+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:47.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:47.597+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:47.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:47.598+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:48.524 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:48.396+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 22880 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:48.525 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:48.175+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:48.525 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:48.175+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:48.525 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:48.175+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:48.525 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:48.175+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:48.712+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:48.712+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:48.712+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:48.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:48.713+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:48.636+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:48.636+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:48.636+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:48.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:48.636+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:49.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:49.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:49.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:49.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:49.191+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:49.742+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:49.742+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:49.742+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:50.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:49.743+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:50.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:49.686+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:50.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:49.686+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:50.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:49.686+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:50.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:49.687+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:50.165+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:50.165+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:50.165+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:50.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:50.165+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:51.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:50.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:50.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:50.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:50.697+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:50.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:51.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:50.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:51.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:50.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:51.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:50.686+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:51.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:51.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:51.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:51.204+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:51.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:51.204+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:51.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:51.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:52.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:51.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:52.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:51.680+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:52.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:51.733+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:52.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:51.733+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:52.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:51.733+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:52.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:51.734+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:52.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:52.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:52.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:52.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:52.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:52.197+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:52.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:52.197+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:52.649+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:52.649+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:52.649+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:52.650+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:53.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:52.718+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:53.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:52.718+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:53.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:52.718+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:53.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:52.719+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:53.522 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:53.398+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 22973 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:53.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:53.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:53.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:53.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:53.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:53.224+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:53.523 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:53.225+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:53.675+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:53.675+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:53.675+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:53.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:53.675+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:53.757+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:53.757+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:53.757+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:53.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:53.757+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:54.470 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:54.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:54.470 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:54.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:54.470 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:54.191+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:54.470 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:54.191+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:54.769 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:54.702+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:54.769 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:54.702+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:54.770 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:54.702+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:54.770 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:54.703+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:55.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:54.768+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:54.768+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:54.768+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:54.769+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:55.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:55.187+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:55.187+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:55.187+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:55.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:55.188+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:56.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:55.725+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:56.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:55.725+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:56.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:55.725+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:56.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:55.726+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:55.813+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:56.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:55.813+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:56.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:55.813+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:56.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:55.814+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:56.476 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:56.163+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:56.476 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:56.163+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:56.476 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:56.163+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:56.476 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:56.165+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:56.817 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:56.697+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:56.817 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:56.697+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:56.817 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:56.697+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:56.817 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:56.698+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:57.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:56.815+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:56.815+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:56.815+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:56.817+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:57.143+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:57.143+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:57.143+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:57.144+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:58.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:57.777+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:58.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:57.777+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:58.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:57.777+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:58.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:57.778+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:58.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:57.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:58.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:57.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:58.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:57.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:58.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:57.672+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:58.402 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:58.105+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:58.403 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:58.105+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:58.403 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:58.105+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:58.403 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:58.105+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:58.668 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:41:58.401+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 23062 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:41:59.103 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:58.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:59.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:58.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:59.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:58.667+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:41:59.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:58.667+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:59.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:58.760+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:59.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:58.760+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:59.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:58.760+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:41:59.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:58.761+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:41:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:59.102+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:59.102+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:59.102+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:41:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:41:59.103+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:00.073 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:59.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:00.073 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:59.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:00.073 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:59.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:00.073 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:41:59.655+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:00.073 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:59.735+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:00.074 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:59.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:00.074 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:59.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:00.074 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:41:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:41:59.737+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:00.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:00.072+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:00.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:00.072+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:00.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:00.072+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:00.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:00.073+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:01.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:00.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:01.041 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:00.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:01.042 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:00.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:01.042 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:00.697+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:01.042 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:00.692+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:01.042 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:00.692+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:01.042 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:00.692+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:01.042 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:00.693+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:01.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:01.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:01.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:01.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:01.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:01.040+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:01.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:01.041+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:02.015 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:01.657+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:02.016 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:01.657+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:02.016 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:01.657+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:02.016 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:01.658+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:02.016 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:01.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:02.017 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:01.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:02.017 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:01.649+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:02.017 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:01.650+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:02.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:02.014+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:02.014+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:02.014+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:02.015+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:03.015 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:02.683+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:03.015 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:02.683+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:03.016 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:02.683+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:03.016 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:02.684+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:03.016 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:02.650+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:03.016 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:02.650+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:03.016 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:02.650+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:03.016 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:02.651+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:03.014+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:03.014+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:03.014+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:03.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:03.016+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:03.674 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:03.403+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 23153 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:03.674 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:03.625+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:03.674 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:03.625+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:03.674 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:03.625+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:03.675 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:03.626+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:04.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:03.673+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:04.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:03.673+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:04.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:03.673+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:04.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:03.674+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:04.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:04.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:04.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:04.035+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:05.020 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:04.692+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:05.021 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:04.692+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:05.021 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:04.692+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:05.021 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:04.693+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:05.021 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:04.666+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:05.021 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:04.666+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:05.021 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:04.666+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:05.022 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:04.667+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:05.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:05.019+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:05.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:05.019+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:05.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:05.019+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:05.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:05.020+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:06.015 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:05.643+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:06.016 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:05.644+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:06.016 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:05.644+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:06.016 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:05.644+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:06.016 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:05.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:06.016 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:05.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:06.017 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:05.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:06.017 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:05.672+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:06.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:06.014+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:06.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:06.014+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:06.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:06.014+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:06.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:06.015+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:06.980 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:06.719+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:06.980 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:06.719+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:06.980 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:06.719+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:06.980 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:06.720+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:06.981 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:06.615+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:06.981 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:06.615+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:06.981 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:06.615+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:06.981 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:06.616+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:07.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:06.979+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:07.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:06.979+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:07.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:06.979+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:07.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:06.980+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:07.975 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:07.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:07.976 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:07.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:07.976 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:07.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:07.976 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:07.744+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:07.976 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:07.628+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:07.977 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:07.628+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:07.977 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:07.628+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:07.977 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:07.628+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:07.974+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:07.974+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:07.974+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:07.975+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:08.664 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:08.406+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 23241 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:09.008 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:08.730+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:09.009 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:08.730+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:09.009 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:08.730+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:09.009 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:08.731+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:09.009 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:08.663+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:09.010 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:08.663+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:09.010 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:08.663+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:09.010 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:08.664+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:09.007+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:09.007+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:09.007+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:09.008+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:10.024 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:09.753+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:10.024 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:09.753+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:10.024 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:09.753+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:10.024 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:09.754+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:10.025 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:09.682+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:10.025 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:09.682+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:10.025 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:09.682+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:10.025 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:09.682+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:10.023+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:10.023+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:10.023+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:10.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:10.024+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:10.979 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:10.768+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:10.979 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:10.768+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:10.980 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:10.768+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:10.980 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:10.769+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:10.980 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:10.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:10.980 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:10.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:10.980 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:10.685+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:10.980 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:10.686+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:10.978+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:10.978+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:10.978+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:10.979+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:12.004 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:11.777+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:12.004 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:11.778+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:12.004 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:11.778+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:12.004 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:11.778+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:12.005 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:11.730+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:12.005 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:11.730+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:12.005 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:11.730+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:12.005 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:11.731+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:12.002+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:12.002+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:12.002+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:12.004+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:13.051 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:12.794+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:13.051 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:12.794+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:13.051 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:12.794+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:13.051 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:12.795+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:13.052 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:12.770+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:13.052 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:12.770+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:13.052 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:12.770+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:13.052 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:12.771+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:13.049+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:13.049+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:13.049+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:13.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:13.051+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:13.785 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:13.408+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 23334 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:13.784+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:13.784+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:13.784+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:13.786+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:14.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:13.826+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:14.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:13.826+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:14.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:13.826+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:14.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:13.826+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:14.020+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:14.020+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:14.020+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:14.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:14.021+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:14.829+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:14.829+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:14.829+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:15.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:14.830+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:15.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:14.786+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:15.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:14.786+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:15.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:14.786+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:15.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:14.787+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:15.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:15.002+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:15.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:15.002+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:15.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:15.002+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:15.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:15.002+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:16.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:15.811+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:16.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:15.811+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:16.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:15.811+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:16.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:15.812+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:15.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:16.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:15.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:16.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:15.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:16.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:15.999+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:15.801+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:15.802+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:15.802+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:15.802+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:17.028+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:17.028+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:17.028+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:17.029+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:17.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:16.837+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:17.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:16.837+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:17.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:16.837+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:17.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:16.838+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:16.785+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:16.785+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:16.785+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:17.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:16.787+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:18.008 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:17.748+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:18.008 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:17.748+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:18.009 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:17.748+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:18.009 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:17.748+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:18.009 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:17.792+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:18.009 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:17.792+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:18.009 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:17.792+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:18.009 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:17.793+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:18.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:18.006+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:18.006+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:18.006+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:18.007+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:18.724 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:18.410+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 23423 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:18.979 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:18.786+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:18.979 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:18.786+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:18.979 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:18.786+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:18.979 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:18.786+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:18.980 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:18.723+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:18.980 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:18.723+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:18.980 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:18.723+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:18.980 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:18.723+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:19.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:18.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:18.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:18.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:19.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:18.978+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:19.746+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:19.746+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:19.746+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:19.747+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:20.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:19.968+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:20.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:19.968+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:20.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:19.968+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:20.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:19.968+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:20.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:19.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:20.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:19.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:20.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:19.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:20.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:19.764+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:21.053 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:21.011+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:21.053 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:21.011+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:21.053 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:21.011+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:21.053 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:21.012+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:21.054 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:20.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:21.054 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:20.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:21.054 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:20.781+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:21.054 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:20.782+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:21.055 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:20.784+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:21.055 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:20.784+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:21.055 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:20.784+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:21.055 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:20.785+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:21.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:21.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:21.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:21.970+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:22.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:21.830+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:21.830+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:21.830+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:21.831+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:22.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:21.745+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:22.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:21.745+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:22.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:21.745+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:22.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:21.746+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:22.707+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:22.707+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:22.707+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:22.708+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:23.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:22.936+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:23.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:22.936+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:23.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:22.936+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:23.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:22.937+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:23.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:22.811+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:23.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:22.811+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:23.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:22.811+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:23.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:22.812+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:23.699 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:23.414+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 23515 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:23.699 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:23.698+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:23.700 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:23.698+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:23.700 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:23.698+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:23.957 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:23.699+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:23.958 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:23.825+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:23.958 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:23.825+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:23.958 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:23.825+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:23.958 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:23.825+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:23.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:23.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:23.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:23.957+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:24.955 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:24.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:24.956 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:24.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:24.956 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:24.671+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:24.956 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:24.671+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:24.956 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:24.820+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:24.957 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:24.820+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:24.957 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:24.820+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:24.957 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:24.820+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:24.954+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:24.954+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:24.954+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:24.956+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:25.958 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:25.648+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:25.958 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:25.648+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:25.958 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:25.648+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:25.959 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:25.649+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:25.959 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:25.852+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:25.959 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:25.852+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:25.959 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:25.852+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:25.959 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:25.853+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:26.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:25.957+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:25.957+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:25.957+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:25.958+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:26.881 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:26.612+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:26.881 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:26.612+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:26.881 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:26.612+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:26.881 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:26.613+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:26.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:26.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:26.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:26.881+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:27.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:26.932+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:27.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:26.932+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:27.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:26.932+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:27.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:26.933+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:27.596+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:27.596+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:27.596+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:27.597+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:27.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:27.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:27.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:27.843+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:27.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:27.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:27.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:27.938+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:28.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:28.416+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 23604 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:28.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:28.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:28.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:28.594+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:28.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:28.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:28.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:29.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:28.938+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:29.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:28.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:29.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:28.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:29.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:28.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:29.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:28.888+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:29.906 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:29.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:29.906 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:29.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:29.906 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:29.614+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:29.907 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:29.614+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:30.192 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:29.904+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:30.192 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:29.904+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:30.192 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:29.904+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:30.192 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:29.905+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:30.193 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:29.907+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:30.193 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:29.907+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:30.193 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:29.907+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:30.193 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:29.908+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:30.921 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:30.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:30.921 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:30.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:30.922 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:30.623+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:30.922 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:30.623+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:30.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:30.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:30.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:30.921+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:30.954+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:31.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:30.954+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:31.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:30.954+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:31.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:30.955+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:31.914 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:31.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:31.915 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:31.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:31.915 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:31.653+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:31.915 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:31.654+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:31.915 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:31.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:31.915 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:31.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:31.915 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:31.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:31.916 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:31.892+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:31.913+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:31.913+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:31.913+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:31.914+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:32.934 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:32.622+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:32.934 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:32.622+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:32.934 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:32.622+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:32.934 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:32.623+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:32.935 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:32.862+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:32.935 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:32.862+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:32.935 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:32.862+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:32.935 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:32.863+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:33.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:32.933+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:33.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:32.933+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:33.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:32.933+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:33.340 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:32.934+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:33.605 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:33.419+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 23696 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:33.877 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:33.652+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:33.878 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:33.652+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:33.878 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:33.652+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:33.878 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:33.653+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:33.899+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:33.899+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:33.899+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:33.900+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:33.876+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:34.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:33.876+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:34.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:33.876+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:34.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:33.877+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:34.871 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:34.608+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:34.871 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:34.608+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:34.871 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:34.608+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:34.871 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:34.609+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:34.913+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:34.913+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:34.913+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:34.914+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:34.870+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:35.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:34.870+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:35.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:34.870+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:35.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:34.871+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:35.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:35.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:35.584+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:35.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:35.585+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:35.904+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:35.904+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:35.904+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:35.905+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:35.904+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:36.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:35.904+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:36.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:35.904+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:36.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:35.904+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:36.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:36.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:36.574+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:36.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:36.575+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:36.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:36.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:37.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:36.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:37.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:36.943+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:36.892+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:36.892+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:36.892+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:37.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:36.893+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:37.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:37.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:37.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:37.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:37.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:37.536+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:37.922+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:37.922+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:38.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:37.922+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:38.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:37.923+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:37.857+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:37.857+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:37.857+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:37.858+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:38.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:38.421+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 23784 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:38.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:38.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:38.527+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:38.527+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:38.920+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:38.920+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:38.920+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:38.921+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:38.901+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:38.901+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:38.901+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:38.902+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:39.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:39.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:39.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:39.510+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:40.193 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:39.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:40.193 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:39.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:40.194 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:39.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:40.194 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:39.875+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:40.194 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:39.943+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:40.195 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:39.943+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:40.195 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:39.943+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:40.195 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:39.943+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:40.842 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:40.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:40.842 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:40.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:40.842 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:40.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:40.842 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:40.465+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:41.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:40.841+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:41.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:40.841+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:41.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:40.841+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:41.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:40.841+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:41.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:40.992+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:41.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:40.992+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:41.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:40.992+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:41.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:40.993+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:41.723 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:41.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:41.723 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:41.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:41.723 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:41.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:41.724 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:41.463+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:42.010 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:41.815+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:42.011 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:41.815+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:42.011 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:41.815+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:42.011 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:41.816+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:42.009+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:42.009+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:42.009+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:42.010+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:42.802 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:42.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:42.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:42.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:42.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:42.487+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:42.824+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:42.824+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:42.824+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:42.824+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:42.988+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:43.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:42.988+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:43.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:42.988+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:43.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:42.989+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:43.783 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:43.424+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 23898 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:43.784 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:43.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:43.784 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:43.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:43.784 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:43.489+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:43.784 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:43.490+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:44.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:43.782+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:44.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:43.782+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:44.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:43.782+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:44.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:43.783+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:44.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:44.025+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:44.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:44.025+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:44.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:44.025+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:44.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:44.026+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:44.757 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:44.452+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:44.758 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:44.452+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:44.758 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:44.452+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:44.758 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:44.453+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:45.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:44.756+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:45.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:44.756+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:45.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:44.756+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:45.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:44.757+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:45.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:44.977+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:45.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:44.977+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:45.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:44.977+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:45.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:44.978+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:45.733 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:45.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:45.733 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:45.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:45.733 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:45.427+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:45.733 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:45.428+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:45.732+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:45.732+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:45.732+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:45.733+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:45.948+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:45.948+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:46.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:45.948+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:46.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:45.948+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:46.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:46.393+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:46.782 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:46.393+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:46.782 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:46.393+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:46.782 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:46.394+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:47.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:46.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:47.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:46.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:47.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:46.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:47.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:46.939+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:46.780+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:47.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:46.780+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:47.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:46.780+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:47.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:46.781+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:47.761 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:47.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:47.761 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:47.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:47.761 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:47.432+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:47.761 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:47.432+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:47.955+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:47.955+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:47.955+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:47.956+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:48.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:47.755+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:48.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:47.755+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:48.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:47.755+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:48.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:47.756+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:48.743 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:48.426+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 23987 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:48.744 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:48.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:48.744 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:48.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:48.744 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:48.464+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:48.744 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:48.464+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:48.742+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:48.742+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:48.742+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:48.743+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:49.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:48.954+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:49.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:48.954+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:49.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:48.954+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:49.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:48.955+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:49.750 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:49.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:49.750 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:49.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:49.750 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:49.418+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:49.750 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:49.418+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:49.749+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:49.749+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:49.749+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:49.750+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:50.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:49.972+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:50.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:49.972+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:50.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:49.972+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:50.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:49.973+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:50.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:50.440+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:50.440+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:50.440+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:50.441+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:51.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:50.976+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:50.976+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:50.977+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:50.977+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:50.740+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:51.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:50.740+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:51.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:50.740+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:51.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:50.740+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:51.750 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:51.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:51.750 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:51.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:51.750 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:51.486+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:51.750 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:51.487+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:52.011 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:51.749+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:52.011 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:51.749+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:52.011 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:51.749+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:52.011 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:51.749+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:52.010+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:52.010+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:52.010+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:52.010+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:52.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:52.700+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:52.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:52.700+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:52.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:52.700+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:52.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:52.701+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:52.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:52.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:52.462+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:52.463+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:52.966+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:52.966+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:52.966+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:52.966+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:53.720 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:53.429+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 24076 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:53.721 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:53.433+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:53.721 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:53.433+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:53.721 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:53.433+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:53.721 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:53.434+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:53.938+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:53.938+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:53.938+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:53.938+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:54.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:53.719+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:54.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:53.719+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:54.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:53.719+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:54.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:53.720+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:54.683 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:54.392+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:54.684 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:54.392+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:54.684 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:54.392+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:54.684 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:54.393+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:54.969 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:54.682+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:54.969 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:54.682+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:54.969 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:54.682+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:54.969 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:54.683+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:55.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:54.968+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:54.968+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:54.968+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:54.969+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:55.651 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:55.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:55.651 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:55.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:55.652 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:55.359+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:55.652 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:55.360+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:55.984 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:55.650+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:55.984 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:55.650+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:55.985 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:55.650+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:55.985 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:55.651+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:55.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:55.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:55.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:56.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:55.984+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:56.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:56.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:56.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:56.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:56.405+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:56.610+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:56.610+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:56.610+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:56.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:56.611+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:56.951+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:56.951+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:56.951+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:56.952+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:57.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:57.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:57.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:57.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:57.383+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:57.616+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:57.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:57.616+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:57.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:57.616+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:57.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:57.617+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:57.994+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:57.994+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:57.994+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:58.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:57.994+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:58.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:42:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:42:58.432+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 24163 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:42:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:58.664+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:58.664+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:58.664+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:42:58.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:58.665+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:58.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:58.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:58.431+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:58.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:58.434+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:59.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:59.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:59.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:42:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:42:59.043+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:42:59.681 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:59.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:59.681 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:59.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:59.681 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:59.404+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:42:59.682 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:42:59.405+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:00.001 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:59.680+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:00.002 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:59.680+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:00.002 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:59.680+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:00.002 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:42:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:42:59.681+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:00.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:00.000+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:00.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:00.000+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:00.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:00.000+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:00.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:00.001+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:00.687+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:00.687+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:00.687+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:00.687+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:00.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:00.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:00.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:00.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:00.451+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:00.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:00.451+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:01.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:01.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:01.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:01.029+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:01.710 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:01.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:01.710 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:01.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:01.710 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:01.424+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:01.710 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:01.425+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:01.991 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:01.708+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:01.991 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:01.708+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:01.991 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:01.708+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:01.991 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:01.708+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:01.989+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:01.989+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:01.989+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:01.990+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:02.455+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:02.455+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:02.455+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:02.456+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:02.659+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:02.659+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:02.659+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:02.660+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:02.968+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:02.968+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:02.968+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:02.969+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:03.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:03.433+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 24254 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:03.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:03.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:03.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:03.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:03.496+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:03.630+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:03.630+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:03.630+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:03.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:03.631+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:03.971+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:03.971+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:03.971+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:04.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:03.972+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:04.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:04.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:04.461+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:04.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:04.462+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:04.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:04.677+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:04.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:04.677+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:04.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:04.677+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:04.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:04.678+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:05.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:04.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:04.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:04.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:04.940+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:05.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:05.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:05.509+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:05.509+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:05.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:05.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:05.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:05.727+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:05.912+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:05.912+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:05.912+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:05.912+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:06.725 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:06.469+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:06.725 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:06.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:06.726 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:06.470+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:06.726 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:06.471+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:06.724+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:06.724+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:06.724+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:07.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:06.725+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:07.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:06.890+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:07.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:06.890+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:07.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:06.890+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:07.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:06.891+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:07.751 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:07.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:07.752 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:07.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:07.752 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:07.476+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:07.752 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:07.477+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:07.857+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:07.857+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:07.857+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:08.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:07.857+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:08.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:07.750+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:08.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:07.750+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:08.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:07.750+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:08.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:07.751+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:08.710 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:08.437+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 24344 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:08.710 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:08.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:08.710 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:08.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:08.711 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:08.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:08.711 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:08.483+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:08.709+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:08.709+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:08.709+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:09.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:08.710+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:09.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:08.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:09.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:08.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:09.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:08.842+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:09.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:08.842+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:09.753 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:09.496+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:09.753 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:09.496+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:09.753 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:09.496+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:09.753 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:09.497+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:09.752+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:09.752+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:09.752+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:09.752+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:10.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:09.874+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:10.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:09.874+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:10.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:09.874+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:10.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:09.874+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:10.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:10.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:10.746 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:10.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:10.746 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:10.472+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:10.746 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:10.473+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:10.744+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:10.744+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:10.744+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:11.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:10.745+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:11.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:10.881+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:11.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:10.881+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:11.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:10.881+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:11.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:10.882+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:11.725 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:11.435+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:11.725 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:11.435+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:11.726 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:11.435+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:11.726 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:11.436+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:11.917+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:11.917+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:11.917+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:12.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:11.918+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:11.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:12.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:11.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:12.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:11.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:12.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:11.728+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:12.736 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:12.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:12.737 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:12.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:12.737 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:12.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:12.737 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:12.396+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:13.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:12.735+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:13.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:12.735+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:13.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:12.735+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:13.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:12.736+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:12.874+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:12.874+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:12.874+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:13.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:12.874+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:13.567 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:13.440+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 24436 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:13.567 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:13.446+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:13.567 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:13.446+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:13.567 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:13.446+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:13.567 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:13.447+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:13.827 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:13.762+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:13.827 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:13.762+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:13.827 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:13.762+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:13.827 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:13.763+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:14.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:13.825+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:13.825+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:13.825+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:13.826+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:14.807 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:14.449+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:14.808 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:14.449+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:14.808 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:14.449+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:14.808 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:14.450+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:14.811+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:14.811+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:14.811+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:14.811+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:15.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:14.806+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:15.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:14.807+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:15.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:14.807+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:15.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:14.807+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:15.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:15.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:15.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:15.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:15.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:15.430+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:15.800 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:15.431+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:15.826+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:15.826+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:15.826+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:15.826+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:16.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:15.799+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:15.799+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:15.799+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:15.800+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:16.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:16.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:16.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:16.474+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:16.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:16.475+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:17.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:16.752+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:17.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:16.752+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:17.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:16.752+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:17.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:16.753+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:17.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:16.821+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:17.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:16.821+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:17.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:16.821+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:17.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:16.821+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:17.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:17.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:17.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:17.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:17.805 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:17.515+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:17.806 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:17.516+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:17.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:17.720+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:17.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:17.720+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:17.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:17.720+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:17.806 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:17.721+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:18.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:17.804+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:18.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:17.804+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:18.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:17.804+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:18.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:17.805+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:18.717 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:18.442+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 24526 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:18.718 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:18.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:18.718 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:18.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:18.718 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:18.522+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:18.718 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:18.523+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:19.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:18.716+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:19.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:18.716+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:19.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:18.716+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:19.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:18.717+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:19.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:18.845+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:19.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:18.845+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:19.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:18.845+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:19.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:18.846+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:19.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:19.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:19.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:19.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:19.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:19.500+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:19.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:19.501+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:19.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:19.678+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:19.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:19.678+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:19.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:19.678+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:19.805 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:19.678+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:20.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:19.802+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:19.802+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:19.802+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:20.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:19.803+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:20.722 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:20.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:20.722 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:20.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:20.723 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:20.457+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:20.723 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:20.458+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:21.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:20.721+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:20.721+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:20.721+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:21.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:20.722+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:20.761+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:20.761+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:20.761+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:21.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:20.762+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:21.716 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:21.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:21.716 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:21.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:21.716 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:21.442+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:21.716 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:21.442+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:21.777+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:21.777+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:21.777+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:22.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:21.778+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:22.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:21.715+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:22.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:21.715+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:21.715+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:22.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:21.715+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:22.729 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:22.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:22.730 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:22.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:22.730 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:22.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:22.730 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:22.418+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:22.793+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:22.793+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:22.794+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:22.795+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:22.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:22.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:23.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:22.728+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:23.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:22.729+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:23.768 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:23.445+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 24617 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:23.769 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:23.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:23.769 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:23.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:23.769 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:23.410+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:23.769 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:23.411+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:24.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:23.824+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:24.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:23.824+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:24.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:23.824+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:24.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:23.825+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:24.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:23.767+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:24.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:23.767+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:24.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:23.767+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:24.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:23.768+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:24.758 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:24.366+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:24.758 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:24.366+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:24.759 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:24.366+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:24.759 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:24.367+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:25.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:24.757+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:25.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:24.757+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:25.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:24.757+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:25.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:24.758+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:24.780+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:24.780+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:25.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:24.780+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:25.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:24.781+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:25.714 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:25.364+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:25.714 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:25.364+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:25.714 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:25.364+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:25.714 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:25.365+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:25.713+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:25.713+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:25.713+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:26.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:25.714+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:26.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:25.736+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:26.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:25.736+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:26.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:25.736+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:26.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:25.737+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:26.718 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:26.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:26.718 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:26.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:26.719 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:26.382+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:26.719 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:26.383+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:26.717+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:26.717+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:26.717+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:26.719+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:26.719+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:27.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:26.719+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:27.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:26.719+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:27.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:26.720+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:27.730 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:27.368+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:27.730 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:27.368+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:27.730 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:27.368+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:27.730 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:27.369+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:27.767+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:27.767+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:27.767+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:28.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:27.768+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:28.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:27.729+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:28.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:27.729+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:28.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:27.729+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:28.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:27.730+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:28.696 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:28.448+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 24709 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:28.697 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:28.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:28.697 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:28.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:28.697 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:28.384+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:28.697 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:28.385+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:28.720+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:28.720+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:28.720+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:29.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:28.721+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:28.696+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:29.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:28.696+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:29.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:28.696+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:29.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:28.697+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:29.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:29.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:29.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:29.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:29.354+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:29.761+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:29.761+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:29.761+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:30.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:29.762+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:30.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:29.712+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:30.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:29.712+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:30.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:29.712+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:30.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:29.713+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:30.368+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:30.368+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:30.368+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:30.368+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:31.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:30.742+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:31.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:30.742+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:31.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:30.742+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:31.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:30.743+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:30.734+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:30.734+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:31.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:30.734+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:31.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:30.735+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:31.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:31.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:31.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:31.353+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:31.354+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:31.776+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:31.776+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:31.776+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:32.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:31.777+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:31.709+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:32.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:31.709+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:32.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:31.709+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:32.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:31.709+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:32.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:32.324+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:32.324+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:32.324+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:32.324+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:32.749+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:32.749+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:32.749+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:33.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:32.750+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:33.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:32.753+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:33.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:32.753+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:33.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:32.753+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:33.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:32.753+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:33.451 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:33.339+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:33.452 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:33.339+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:33.452 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:33.339+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:33.452 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:33.340+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:33.748 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:33.450+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 24801 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:33.747+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:33.747+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:33.747+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:34.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:33.748+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:34.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:33.799+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:34.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:33.799+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:34.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:33.799+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:34.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:33.799+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:34.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:34.310+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:34.310+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:34.310+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:34.311+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:34.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:34.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:34.763+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:35.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:34.763+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:35.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:34.752+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:35.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:34.752+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:35.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:34.752+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:35.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:34.753+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:35.733 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:35.354+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:35.733 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:35.354+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:35.733 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:35.354+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:35.733 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:35.354+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:36.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:35.760+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:35.760+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:35.760+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:35.761+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:36.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:35.732+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:36.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:35.732+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:36.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:35.732+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:36.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:35.733+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:36.690 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:36.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:36.690 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:36.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:36.690 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:36.395+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:36.690 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:36.396+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:37.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:36.736+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:37.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:36.736+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:37.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:36.736+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:37.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:36.736+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:37.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:36.689+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:37.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:36.689+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:37.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:36.689+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:37.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:36.690+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:37.675 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:37.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:37.675 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:37.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:37.676 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:37.374+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:37.676 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:37.375+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:38.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:37.689+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:38.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:37.689+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:38.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:37.689+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:38.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:37.690+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:38.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:37.674+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:38.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:37.674+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:38.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:37.674+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:38.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:37.675+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:38.654 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:38.453+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 24891 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:38.655 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:38.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:38.655 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:38.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:38.655 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:38.357+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:38.655 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:38.358+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:39.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:38.684+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:39.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:38.684+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:39.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:38.684+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:39.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:38.685+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:39.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:38.653+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:39.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:38.653+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:39.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:38.653+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:39.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:38.655+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:39.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:39.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:39.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:39.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:39.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:39.312+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:39.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:39.313+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:40.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:39.698+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:39.698+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:39.698+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:39.699+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:40.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:39.655+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:40.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:39.655+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:40.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:39.655+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:40.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:39.655+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:40.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:40.305+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:40.305+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:40.305+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:40.306+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:40.924 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:40.652+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:40.924 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:40.652+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:40.924 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:40.652+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:40.924 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:40.653+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:40.925 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:40.731+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:40.925 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:40.731+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:40.925 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:40.731+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:40.925 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:40.731+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:41.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:41.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:41.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:41.345+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:41.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:41.346+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:42.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:41.690+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:42.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:41.690+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:42.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:41.690+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:42.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:41.691+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:42.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:41.643+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:42.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:41.643+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:42.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:41.643+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:42.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:41.643+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:42.317+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:42.317+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:42.317+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:42.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:42.318+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:43.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:42.733+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:42.733+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:42.733+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:43.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:42.733+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:42.630+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:42.630+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:43.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:42.630+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:43.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:42.631+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:43.624 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:43.456+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 24983 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:43.625 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:43.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:43.625 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:43.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:43.625 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:43.365+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:43.625 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:43.366+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:43.996 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:43.632+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:43.996 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:43.632+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:43.996 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:43.632+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:43.997 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:43.633+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:43.997 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:43.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:43.997 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:43.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:43.997 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:43.736+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:43.997 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:43.737+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:44.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:44.328+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:44.328+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:44.328+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:44.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:44.329+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:45.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:44.622+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:45.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:44.622+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:45.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:44.622+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:45.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:44.623+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:45.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:44.737+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:45.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:44.737+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:45.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:44.737+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:45.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:44.738+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:45.622 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:45.367+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:45.623 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:45.367+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:45.623 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:45.367+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:45.623 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:45.368+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:46.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:45.765+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:45.765+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:45.765+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:45.765+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:45.621+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:45.621+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:45.621+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:46.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:45.622+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:46.780 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:46.622+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:46.781 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:46.622+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:46.781 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:46.622+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:46.781 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:46.622+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:46.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:46.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:46.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:46.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:46.781 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:46.397+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:46.782 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:46.398+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:47.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:46.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:47.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:46.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:47.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:46.779+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:47.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:46.780+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:47.794 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:47.611+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:47.795 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:47.611+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:47.795 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:47.611+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:47.795 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:47.612+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:47.795 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:47.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:47.795 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:47.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:47.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:47.415+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:47.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:47.416+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:48.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:47.792+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:47.792+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:47.792+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:47.793+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:48.837 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:48.457+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 25069 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:48.837 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:48.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:48.837 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:48.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:48.838 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:48.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:48.838 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:48.463+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:48.838 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:48.615+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:48.838 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:48.615+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:48.838 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:48.615+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:48.838 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:48.615+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:49.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:48.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:49.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:48.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:49.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:48.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:49.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:48.836+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:49.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:49.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:49.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:49.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:49.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:49.417+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:49.796 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:49.418+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:49.797 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:49.590+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:49.797 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:49.590+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:49.797 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:49.590+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:49.797 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:49.591+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:50.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:49.794+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:50.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:49.794+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:50.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:49.794+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:50.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:49.795+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:50.552 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:50.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:50.552 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:50.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:50.552 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:50.439+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:50.552 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:50.440+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:50.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:50.550+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:50.550+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:50.550+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:50.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:50.551+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:50.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:50.773+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:50.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:50.773+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:50.773+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:50.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:50.774+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:51.728 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:51.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:51.728 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:51.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:51.728 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:51.447+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:51.729 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:51.448+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:51.729 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:51.506+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:51.729 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:51.506+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:51.729 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:51.506+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:51.729 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:51.506+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:52.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:51.792+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:52.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:51.792+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:52.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:51.792+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:52.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:51.792+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:52.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:52.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:52.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:52.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:52.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:52.481+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:52.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:52.511+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:52.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:52.511+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:52.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:52.511+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:52.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:52.512+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:52.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:52.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:52.835+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:53.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:52.836+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:53.472 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:53.461+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 25160 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:53.792 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:53.471+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:53.792 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:53.471+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:53.792 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:53.471+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:53.793 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:53.472+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:53.793 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:53.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:53.793 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:53.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:53.793 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:53.499+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:53.793 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:53.500+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:53.791+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:53.791+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:53.791+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:54.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:53.792+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:54.744 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:54.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:54.744 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:54.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:54.744 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:54.432+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:54.745 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:54.432+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:54.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:54.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:54.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:54.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:54.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:54.463+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:54.746 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:54.463+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:55.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:54.743+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:54.743+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:54.743+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:55.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:54.744+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:55.777 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:55.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:55.777 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:55.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:55.778 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:55.505+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:55.778 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:55.506+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:55.778 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:55.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:55.778 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:55.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:55.778 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:55.421+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:55.778 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:55.422+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:55.776+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:56.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:55.776+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:56.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:55.776+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:56.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:55.777+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:56.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:56.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:56.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:56.525+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:56.526+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:56.396+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:56.396+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:56.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:56.396+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:56.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:56.397+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:56.752+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:56.752+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:56.752+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:57.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:56.753+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:57.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:57.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:57.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:57.523+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:57.524+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:57.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:57.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:57.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:57.351+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:57.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:57.352+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:58.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:57.801+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:58.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:57.801+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:58.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:57.801+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:58.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:57.802+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:58.813 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:43:58.464+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 25251 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:43:58.814 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:58.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:58.814 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:58.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:58.814 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:58.530+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:58.814 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:58.531+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:58.815 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:58.396+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:58.815 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:58.396+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:58.815 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:58.396+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:58.815 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:58.397+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:59.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:58.812+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:59.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:58.812+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:59.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:58.813+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:43:59.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:58.813+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:59.831 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:59.533+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:59.831 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:59.533+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:59.831 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:59.533+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:43:59.831 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:43:59.534+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:43:59.831 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:59.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:59.831 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:59.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:59.832 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:59.424+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:43:59.832 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:43:59.425+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:00.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:59.829+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:00.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:59.829+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:00.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:59.829+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:00.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:43:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:43:59.830+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:00.520 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:00.382+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:00.520 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:00.382+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:00.520 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:00.382+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:00.520 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:00.383+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:00.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:00.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:00.519+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:00.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:00.520+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:00.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:00.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:00.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:00.881+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:01.729 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:01.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:01.729 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:01.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:01.729 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:01.398+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:01.729 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:01.399+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:01.730 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:01.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:01.730 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:01.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:01.730 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:01.542+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:01.730 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:01.542+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:01.899+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:01.899+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:01.899+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:02.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:01.900+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:02.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:02.540+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:02.540+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:02.540+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:02.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:02.541+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:02.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:02.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:02.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:02.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:02.358+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:02.805 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:02.359+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:02.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:02.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:02.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:02.865+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:03.625 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:03.467+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 25341 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:03.626 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:03.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:03.626 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:03.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:03.626 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:03.369+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:03.626 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:03.370+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:03.627 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:03.554+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:03.627 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:03.554+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:03.627 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:03.554+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:03.627 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:03.555+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:04.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:03.830+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:04.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:03.830+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:04.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:03.830+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:04.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:03.831+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:04.819 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:04.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:04.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:04.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:04.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:04.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:04.820 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:04.507+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:04.820 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:04.367+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:04.821 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:04.367+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:04.821 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:04.367+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:04.821 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:04.368+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:04.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:04.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:04.818+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:05.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:04.819+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:05.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:05.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:05.482+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:05.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:05.483+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:05.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:05.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:05.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:05.384+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:05.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:05.384+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:06.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:05.856+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:05.856+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:05.856+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:05.857+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:06.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:06.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:06.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:06.502+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:06.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:06.503+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:06.385+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:06.385+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:06.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:06.385+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:06.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:06.386+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:06.863+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:06.864+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:06.864+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:06.865+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:07.847 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:07.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:07.847 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:07.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:07.847 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:07.480+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:07.847 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:07.481+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:07.848 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:07.413+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:07.848 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:07.413+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:07.848 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:07.413+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:07.848 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:07.413+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:07.848 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:07.846+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:07.848 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:07.846+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:07.849 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:07.846+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:08.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:07.847+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:08.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:08.469+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 25432 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:08.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:08.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:08.430+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:08.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:08.431+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:08.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:08.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:08.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:08.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:08.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:08.495+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:08.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:08.495+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:08.866+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:08.866+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:08.866+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:09.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:08.867+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:09.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:09.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:09.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:09.492+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:09.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:09.493+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:09.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:09.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:09.475+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:09.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:09.476+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:10.201 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:09.904+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:10.201 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:09.904+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:10.201 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:09.904+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:10.201 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:09.905+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:10.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:10.511+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:10.511+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:10.511+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:10.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:10.512+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:10.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:10.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:10.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:10.507+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:10.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:10.508+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:10.908+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:10.908+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:10.908+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:10.909+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:11.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:11.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:11.536+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:11.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:11.537+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:11.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:11.551+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:11.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:11.551+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:11.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:11.551+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:11.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:11.552+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:12.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:11.902+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:12.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:11.902+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:12.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:11.902+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:12.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:11.903+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:12.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:12.510+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:12.803 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:12.510+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:12.510+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:12.804 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:12.511+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:12.539+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:12.539+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:12.539+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:12.804 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:12.540+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:13.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:12.888+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:13.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:12.888+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:13.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:12.888+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:13.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:12.889+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:13.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:13.472+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 25523 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:13.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:13.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:13.547+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:13.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:13.548+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:13.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:13.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:13.524+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:13.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:13.525+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:14.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:13.914+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:14.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:13.914+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:14.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:13.914+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:14.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:13.915+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:14.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:14.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:14.593+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:14.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:14.593+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:14.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:14.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:14.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:14.533+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:14.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:14.534+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:15.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:14.933+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:15.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:14.933+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:15.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:14.933+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:15.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:14.934+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:15.505+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:15.505+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:15.505+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:15.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:15.506+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:15.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:15.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:15.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:15.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:15.625+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:15.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:15.626+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:16.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:15.906+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:15.906+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:15.906+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:16.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:15.907+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:16.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:16.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:16.637+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:16.638+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:16.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:16.526+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:16.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:16.526+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:16.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:16.526+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:16.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:16.527+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:16.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:16.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:16.891+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:17.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:16.891+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:17.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:17.547+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:17.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:17.547+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:17.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:17.547+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:17.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:17.548+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:17.685+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:17.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:17.685+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:17.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:17.685+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:17.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:17.686+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:18.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:17.892+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:17.892+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:17.892+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:18.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:17.893+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:18.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:18.475+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 25614 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:18.572+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:18.572+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:18.572+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:18.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:18.573+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:18.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:18.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:18.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:18.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:18.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:18.654+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:18.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:18.655+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:18.876+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:18.876+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:18.876+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:18.877+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:19.645+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:19.645+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:19.645+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:19.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:19.646+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:19.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:19.599+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:19.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:19.599+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:19.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:19.599+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:19.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:19.599+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:20.202 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:19.923+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:20.202 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:19.923+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:20.202 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:19.923+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:20.202 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:19.924+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:20.925 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:20.634+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:20.926 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:20.634+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:20.926 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:20.634+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:20.926 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:20.635+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:20.926 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:20.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:20.926 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:20.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:20.926 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:20.696+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:20.927 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:20.697+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:21.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:20.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:20.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:20.924+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:20.925+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:21.927 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:21.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:21.927 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:21.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:21.928 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:21.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:21.928 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:21.645+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:21.928 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:21.675+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:21.928 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:21.675+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:21.928 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:21.675+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:21.928 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:21.676+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:21.926+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:21.926+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:21.926+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:22.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:21.927+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:22.961 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:22.659+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:22.961 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:22.659+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:22.961 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:22.659+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:22.962 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:22.661+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:22.962 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:22.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:22.962 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:22.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:22.962 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:22.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:22.962 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:22.660+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:23.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:22.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:23.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:22.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:23.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:22.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:23.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:22.961+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:23.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:23.477+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 25704 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:23.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:23.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:23.618+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:23.618+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:23.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:23.706+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:23.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:23.706+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:23.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:23.706+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:23.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:23.706+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:23.977+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:23.977+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:23.977+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:23.978+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:24.962 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:24.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:24.962 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:24.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:24.962 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:24.629+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:24.962 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:24.629+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:24.962 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:24.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:24.962 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:24.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:24.963 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:24.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:24.963 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:24.738+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:24.963 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:24.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:24.963 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:24.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:24.963 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:24.960+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:25.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:24.962+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:25.596+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:25.597+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:25.597+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:25.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:25.598+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:25.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:25.717+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:25.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:25.717+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:25.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:25.717+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:25.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:25.718+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:26.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:25.931+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:25.931+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:25.931+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:25.932+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:26.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:26.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:26.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:26.570+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:26.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:26.572+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:26.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:26.684+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:26.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:26.684+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:26.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:26.684+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:26.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:26.684+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:26.964+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:26.964+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:27.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:26.964+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:27.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:26.965+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:27.986 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:27.699+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:27.986 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:27.699+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:27.986 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:27.699+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:27.987 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:27.700+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:27.987 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:27.620+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:27.987 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:27.620+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:27.987 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:27.620+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:27.987 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:27.621+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:28.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:27.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:28.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:27.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:28.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:27.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:28.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:27.986+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:28.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:28.480+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 25795 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:28.681+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:28.681+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:28.681+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:28.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:28.682+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:28.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:28.652+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:28.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:28.652+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:28.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:28.652+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:28.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:28.652+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:28.986+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:28.986+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:28.986+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:29.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:28.987+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:29.942 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:29.653+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:29.942 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:29.653+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:29.942 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:29.653+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:29.942 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:29.654+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:29.942 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:29.666+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:29.943 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:29.666+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:29.943 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:29.666+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:29.943 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:29.667+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:30.203 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:29.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:30.203 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:29.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:30.203 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:29.941+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:30.203 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:29.941+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:30.954 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:30.641+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:30.955 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:30.641+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:30.955 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:30.641+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:30.955 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:30.642+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:30.955 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:30.681+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:30.955 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:30.681+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:30.956 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:30.681+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:30.956 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:30.682+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:31.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:30.953+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:30.953+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:30.953+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:31.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:30.954+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:31.912 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:31.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:31.912 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:31.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:31.913 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:31.630+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:31.913 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:31.631+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:31.913 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:31.709+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:31.913 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:31.709+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:31.913 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:31.709+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:31.913 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:31.710+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:32.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:31.911+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:31.911+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:31.911+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:32.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:31.912+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:32.908 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:32.604+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:32.908 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:32.604+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:32.908 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:32.604+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:32.909 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:32.605+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:32.909 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:32.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:32.909 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:32.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:32.909 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:32.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:32.909 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:32.728+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:33.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:32.907+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:33.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:32.907+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:33.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:32.907+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:33.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:32.908+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:33.607 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:33.482+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 25886 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:33.941 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:33.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:33.941 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:33.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:33.941 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:33.650+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:33.941 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:33.651+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:33.942 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:33.686+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:33.942 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:33.686+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:33.942 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:33.686+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:33.942 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:33.687+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:34.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:33.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:33.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:33.939+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:33.940+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:34.975 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:34.698+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:34.975 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:34.698+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:34.976 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:34.698+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:34.976 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:34.700+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:34.976 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:34.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:34.976 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:34.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:34.976 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:34.663+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:34.976 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:34.664+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:34.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:34.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:34.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:35.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:34.974+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:35.973 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:35.722+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:35.973 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:35.722+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:35.974 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:35.722+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:35.974 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:35.723+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:35.974 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:35.714+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:35.974 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:35.714+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:35.974 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:35.714+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:35.974 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:35.715+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:35.972+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:35.972+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:35.972+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:36.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:35.972+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:36.939 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:36.725+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:36.939 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:36.725+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:36.939 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:36.725+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:36.939 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:36.725+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:36.939 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:36.669+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:36.940 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:36.669+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:36.940 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:36.669+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:36.940 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:36.669+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:37.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:36.938+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:36.938+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:36.938+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:37.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:36.938+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:37.947 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:37.661+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:37.948 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:37.661+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:37.948 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:37.661+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:37.948 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:37.662+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:37.948 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:37.683+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:37.948 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:37.683+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:37.948 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:37.683+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:37.948 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:37.683+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:38.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:37.946+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:37.946+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:37.946+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:37.947+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:38.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:38.486+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 25977 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:38.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:38.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:38.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:38.655+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:38.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:38.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:38.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:38.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:38.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:38.644+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:38.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:38.644+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:39.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:38.969+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:38.969+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:38.969+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:38.970+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:39.955 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:39.954+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:39.955 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:39.954+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:39.955 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:39.664+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:39.955 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:39.664+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:39.956 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:39.664+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:39.956 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:39.665+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:39.956 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:39.670+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:39.956 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:39.670+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:39.956 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:39.670+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:39.956 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:39.670+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:40.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:39.954+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:40.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:39.955+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:40.981 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:40.700+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:40.982 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:40.700+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:40.982 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:40.700+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:40.982 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:40.701+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:40.982 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:40.710+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:40.982 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:40.710+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:40.982 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:40.710+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:40.983 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:40.710+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:41.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:40.980+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:40.980+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:40.980+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:40.981+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:42.030 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:41.729+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:42.030 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:41.729+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:42.030 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:41.729+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:42.030 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:41.730+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:42.031 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:41.751+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:42.031 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:41.751+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:42.031 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:41.751+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:42.031 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:41.751+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:42.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:42.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:42.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:42.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:42.030+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:43.072 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:42.768+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:43.072 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:42.768+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:43.072 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:42.768+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:43.072 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:42.768+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:43.073 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:42.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:43.073 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:42.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:43.073 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:42.727+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:43.073 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:42.728+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:43.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:43.068+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:43.068+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:43.068+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:43.069+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:43.775 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:43.489+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 26066 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:44.030 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:43.775+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:44.031 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:43.775+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:44.031 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:43.775+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:44.031 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:43.776+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:44.031 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:43.810+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:44.031 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:43.810+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:44.031 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:43.810+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:44.032 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:43.810+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:44.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:44.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:44.029+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:44.030+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:45.024 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:44.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:45.024 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:44.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:45.024 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:44.737+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:45.024 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:44.738+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:45.025 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:44.796+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:45.025 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:44.796+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:45.025 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:44.796+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:45.025 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:44.797+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:45.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:45.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:45.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:45.023+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:45.024+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:46.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:45.981+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:45.981+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:45.981+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:46.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:45.982+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:45.813+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:45.813+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:46.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:45.813+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:46.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:45.814+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:46.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:45.766+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:46.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:45.766+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:46.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:45.766+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:46.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:45.767+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:46.799+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:46.799+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:46.799+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:46.800+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:47.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:47.019+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:47.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:47.019+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:47.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:47.019+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:47.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:47.020+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:47.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:46.859+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:47.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:46.859+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:47.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:46.859+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:47.107 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:46.860+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:47.777+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:47.777+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:47.777+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:47.778+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:48.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:48.013+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:48.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:48.013+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:48.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:48.013+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:48.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:48.014+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:48.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:47.884+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:48.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:47.884+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:48.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:47.884+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:48.107 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:47.885+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:48.813 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:48.492+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 26157 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:49.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:48.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:48.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:48.813+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:48.813+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:48.905+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:48.905+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:49.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:48.905+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:49.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:48.905+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:49.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:48.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:49.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:48.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:49.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:48.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:49.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:48.984+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:49.837+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:49.837+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:49.837+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:49.838+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:50.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:49.955+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:50.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:49.955+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:50.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:49.955+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:50.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:49.956+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:50.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:49.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:50.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:49.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:50.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:49.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:50.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:49.986+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:51.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:50.842+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:50.842+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:50.842+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:50.843+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:51.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:51.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:51.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:51.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:51.028+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:51.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:51.029+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:51.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:50.973+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:51.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:50.973+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:51.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:50.973+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:51.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:50.974+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:51.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:51.820+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:51.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:51.820+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:51.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:51.820+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:51.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:51.821+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:52.015+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:52.015+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:52.015+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:52.016+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:52.036+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:52.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:52.036+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:52.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:52.036+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:52.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:52.037+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:53.005+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:53.005+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:53.005+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:53.005+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:53.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:52.862+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:53.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:52.862+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:53.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:52.862+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:53.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:52.863+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:53.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:53.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:53.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:53.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:53.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:53.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:53.108 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:53.033+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:53.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:53.494+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 26247 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:54.012+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:54.012+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:54.012+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:54.013+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:54.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:54.014+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:54.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:54.014+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:54.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:54.014+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:54.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:54.015+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:54.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:53.860+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:54.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:53.860+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:54.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:53.860+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:54.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:53.861+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:55.038 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:54.893+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:55.038 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:54.893+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:55.039 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:54.893+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:55.039 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:54.893+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:55.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:55.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:55.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:55.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:55.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:55.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:55.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:55.038+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:55.052+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:55.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:55.052+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:55.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:55.052+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:55.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:55.053+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:55.854+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:55.854+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:55.854+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:55.854+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:56.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:55.992+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:56.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:55.992+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:56.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:55.992+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:56.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:55.993+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:56.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:56.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:56.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:56.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:56.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:56.056+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:56.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:56.057+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:57.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:57.027+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:57.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:57.027+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:57.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:57.027+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:57.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:57.028+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:57.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:56.852+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:57.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:56.852+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:57.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:56.852+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:57.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:56.852+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:57.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:57.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:57.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:57.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:57.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:57.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:57.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:57.094+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:58.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:57.833+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:58.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:57.833+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:58.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:57.833+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:58.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:57.834+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:58.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:58.052+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:58.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:58.052+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:58.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:58.052+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:58.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:58.053+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:58.498 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:58.104+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:58.499 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:58.104+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:58.499 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:58.104+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:58.499 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:58.105+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:58.802 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:44:58.497+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 26338 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:44:59.080 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:58.801+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:59.080 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:58.801+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:59.080 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:58.801+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:44:59.081 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:58.802+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:59.081 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:59.036+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:59.081 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:59.036+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:59.081 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:59.036+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:44:59.081 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:44:59.036+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:44:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:59.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:59.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:59.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:44:59.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:44:59.080+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:00.049 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:59.773+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:00.050 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:59.773+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:00.050 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:59.773+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:00.050 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:44:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:44:59.774+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:00.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:00.048+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:00.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:00.048+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:00.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:00.048+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:00.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:00.049+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:00.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:00.113+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:00.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:00.113+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:00.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:00.113+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:00.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:00.114+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:01.094 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:01.016+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:01.095 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:01.016+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:01.095 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:01.016+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:01.095 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:01.017+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:01.095 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:00.784+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:01.095 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:00.784+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:01.095 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:00.784+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:01.095 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:00.784+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:01.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:01.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:01.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:01.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:01.093+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:01.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:01.094+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:02.006 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:01.747+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:02.007 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:01.747+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:02.007 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:01.747+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:02.007 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:01.748+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:02.005+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:02.005+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:02.005+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:02.006+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:02.131+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:02.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:02.131+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:02.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:02.131+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:02.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:02.132+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:02.779+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:02.779+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:02.779+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:03.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:02.780+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:03.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:02.987+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:03.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:02.987+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:03.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:02.987+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:03.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:02.988+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:03.501 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:03.149+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:03.501 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:03.149+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:03.502 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:03.149+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:03.502 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:03.150+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:03.776 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:03.500+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 26428 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:04.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:04.003+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:04.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:04.003+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:04.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:04.003+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:04.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:04.004+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:03.775+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:04.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:03.775+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:04.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:03.775+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:04.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:03.775+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:04.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:04.114+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:04.114+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:04.114+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:04.115+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:05.072 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:05.022+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:05.072 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:05.022+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:05.072 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:05.022+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:05.072 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:05.023+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:05.072 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:04.777+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:05.073 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:04.777+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:05.073 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:04.777+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:05.073 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:04.777+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:05.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:05.071+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:05.071+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:05.071+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:05.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:05.071+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:06.025 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:05.769+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:06.026 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:05.769+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:06.026 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:05.769+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:06.026 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:05.770+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:06.024+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:06.025+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:06.025+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:06.026+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:06.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:06.045+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:06.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:06.045+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:06.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:06.045+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:06.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:06.045+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:07.067 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:06.797+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:07.067 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:06.797+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:07.068 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:06.797+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:07.068 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:06.798+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:07.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:07.074+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:07.074+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:07.074+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:07.075+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:07.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:07.066+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:07.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:07.066+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:07.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:07.066+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:07.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:07.067+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:08.102 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:07.825+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:08.102 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:07.825+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:08.102 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:07.825+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:08.103 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:07.826+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:08.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:08.101+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:08.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:08.101+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:08.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:08.101+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:08.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:08.102+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:08.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:08.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:08.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:08.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:08.103+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:08.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:08.104+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:08.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:08.503+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 26518 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:09.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:09.052+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:09.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:09.052+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:09.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:09.052+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:09.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:09.053+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:08.867+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:09.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:08.867+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:09.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:08.867+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:09.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:08.868+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:09.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:09.068+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:09.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:09.068+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:09.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:09.068+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:09.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:09.068+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:10.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:10.014+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:10.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:10.014+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:10.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:10.014+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:10.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:10.015+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:09.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:10.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:09.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:10.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:09.829+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:10.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:09.830+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:10.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:10.073+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:10.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:10.073+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:10.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:10.073+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:10.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:10.074+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:11.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:11.000+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:11.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:11.000+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:11.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:11.000+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:11.098 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:11.001+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:11.099 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:10.794+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:11.099 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:10.794+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:11.099 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:10.794+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:11.099 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:10.795+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:11.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:11.097+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:11.097+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:11.097+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:11.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:11.097+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:12.003 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:11.820+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:12.003 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:11.820+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:12.004 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:11.820+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:12.004 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:11.821+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:12.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:12.002+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:12.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:12.002+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:12.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:12.002+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:12.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:12.003+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:12.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:12.058+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:12.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:12.058+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:12.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:12.058+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:12.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:12.058+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:13.037 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:12.985+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:13.037 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:12.985+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:13.037 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:12.985+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:13.037 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:12.986+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:13.038 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:12.775+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:13.038 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:12.775+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:13.038 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:12.775+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:13.038 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:12.775+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:13.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:13.036+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:13.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:13.036+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:13.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:13.036+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:13.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:13.037+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:13.769 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:13.506+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 26609 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:14.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:13.953+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:14.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:13.953+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:14.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:13.953+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:14.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:13.954+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:14.016+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:14.016+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:14.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:14.016+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:14.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:14.017+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:13.768+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:13.768+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:13.768+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:14.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:13.769+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:15.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:14.796+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:14.796+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:14.796+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:15.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:14.797+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:15.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:14.995+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:15.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:14.995+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:15.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:14.995+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:15.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:14.996+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:15.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:15.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:15.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:15.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:15.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:15.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:15.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:15.019+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:15.789+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:15.789+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:15.789+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:16.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:15.790+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:16.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:16.009+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:16.009+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:16.009+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:16.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:16.010+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:16.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:16.023+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:16.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:16.023+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:16.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:16.023+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:16.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:16.023+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:16.815+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:16.815+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:16.815+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:17.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:16.816+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:17.006+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:17.006+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:17.006+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:17.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:17.006+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:17.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:17.043+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:17.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:17.043+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:17.107 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:17.043+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:17.107 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:17.044+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:18.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:17.828+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:18.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:17.828+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:18.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:17.828+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:18.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:17.828+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:18.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:18.047+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:18.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:18.047+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:18.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:18.047+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:18.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:18.048+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:18.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:18.044+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:18.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:18.044+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:18.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:18.044+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:18.107 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:18.045+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:18.793 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:18.509+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 26699 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:19.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:19.012+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:19.012+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:19.012+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:19.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:19.013+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:19.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:18.791+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:19.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:18.791+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:19.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:18.791+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:19.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:18.792+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:19.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:18.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:19.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:18.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:19.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:18.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:19.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:19.000+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:19.972+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:20.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:19.972+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:20.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:19.972+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:20.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:19.973+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:20.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:19.760+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:20.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:19.760+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:20.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:19.760+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:20.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:19.760+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:20.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:20.003+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:20.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:20.003+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:20.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:20.003+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:20.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:20.004+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:21.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:20.977+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:21.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:20.977+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:21.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:20.977+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:21.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:20.978+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:20.804+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:21.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:20.804+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:21.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:20.804+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:21.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:20.805+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:21.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:21.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:21.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:21.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:21.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:21.018+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:21.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:21.019+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:22.034 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:21.939+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:22.034 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:21.939+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:22.034 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:21.939+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:22.034 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:21.939+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:22.035 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:21.803+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:22.035 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:21.803+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:22.035 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:21.803+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:22.035 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:21.803+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:22.300 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:22.033+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:22.300 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:22.033+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:22.300 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:22.033+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:22.300 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:22.033+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:23.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:23.003+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:23.003+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:23.003+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:23.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:23.004+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:22.948+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:23.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:22.948+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:23.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:22.948+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:23.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:22.948+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:23.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:22.809+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:23.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:22.809+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:23.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:22.809+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:23.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:22.809+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:23.815 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:23.512+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 26789 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:24.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:23.814+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:24.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:23.814+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:24.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:23.814+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:24.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:23.815+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:24.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:23.919+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:24.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:23.919+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:24.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:23.919+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:24.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:23.919+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:24.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:24.054+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:24.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:24.054+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:24.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:24.054+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:24.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:24.055+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:24.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:24.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:24.851+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:25.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:24.852+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:25.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:24.873+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:25.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:24.873+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:25.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:24.873+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:25.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:24.874+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:25.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:25.084+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:25.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:25.084+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:25.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:25.084+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:25.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:25.084+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:26.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:25.875+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:26.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:25.875+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:26.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:25.875+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:26.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:25.876+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:26.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:25.840+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:26.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:25.840+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:26.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:25.840+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:26.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:25.841+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:26.123+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:26.123+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:26.123+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:26.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:26.124+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:27.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:26.834+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:26.834+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:26.834+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:27.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:26.834+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:26.915+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:27.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:26.915+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:27.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:26.915+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:27.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:26.916+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:27.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:27.113+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:27.113+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:27.113+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:27.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:27.114+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:28.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:28.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:28.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:28.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:28.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:28.107+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:28.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:28.108+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:28.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:27.864+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:28.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:27.864+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:28.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:27.864+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:28.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:27.865+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:28.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:27.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:28.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:27.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:28.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:27.956+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:28.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:27.957+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:28.845 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:28.515+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 26880 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:29.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:28.845+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:29.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:28.845+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:29.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:28.845+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:29.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:28.845+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:29.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:28.993+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:29.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:28.993+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:29.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:28.993+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:29.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:28.994+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:29.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:29.133+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:29.133+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:29.133+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:29.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:29.133+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:30.128 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:29.873+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:30.129 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:29.873+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:30.129 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:29.873+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:30.129 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:29.874+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:30.129 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:29.952+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:30.129 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:29.952+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:30.130 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:29.952+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:30.130 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:29.954+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:30.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:30.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:30.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:30.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:30.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:30.127+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:30.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:30.129+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:31.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:30.896+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:31.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:30.897+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:31.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:30.897+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:31.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:30.898+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:31.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:30.936+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:31.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:30.936+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:31.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:30.936+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:31.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:30.936+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:31.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:31.085+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:31.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:31.085+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:31.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:31.085+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:31.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:31.086+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:32.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:31.904+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:32.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:31.904+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:32.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:31.904+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:32.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:31.905+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:31.971+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:31.971+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:31.971+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:32.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:31.971+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:32.088+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:32.088+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:32.088+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:32.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:32.089+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:33.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:32.940+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:33.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:32.940+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:33.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:32.941+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:33.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:32.942+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:32.957+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:32.957+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:32.957+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:33.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:32.958+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:33.519 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:33.135+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:33.519 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:33.135+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:33.519 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:33.135+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:33.520 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:33.136+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:33.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:33.518+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 26971 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:34.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:33.912+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:33.912+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:33.912+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:33.912+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:34.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:34.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:34.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:34.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:34.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:34.094+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:34.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:34.094+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:34.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:33.895+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:34.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:33.895+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:34.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:33.895+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:34.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:33.896+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:35.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:34.894+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:34.894+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:34.894+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:34.895+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:35.047+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:35.047+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:35.047+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:35.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:35.048+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:35.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:34.936+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:35.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:34.936+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:35.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:34.936+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:35.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:34.937+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:36.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:35.937+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:36.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:35.937+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:36.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:35.937+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:36.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:35.938+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:35.932+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:36.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:35.932+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:36.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:35.932+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:36.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:35.933+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:36.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:35.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:36.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:35.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:36.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:35.999+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:36.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:36.000+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:37.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:36.971+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:37.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:36.971+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:37.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:36.971+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:37.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:36.972+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:36.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:36.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:36.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:37.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:36.950+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:37.041+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:37.041+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:37.041+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:37.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:37.042+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:38.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:37.917+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:37.917+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:37.917+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:37.918+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:38.015+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:38.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:38.015+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:38.015+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:38.016+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:38.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:37.972+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:38.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:37.972+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:38.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:37.972+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:38.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:37.973+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:38.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:38.520+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 27062 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:38.930+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:38.930+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:38.930+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:38.930+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:39.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:38.930+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:39.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:38.930+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:39.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:38.930+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:39.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:38.930+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:38.980+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:38.980+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:38.980+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:39.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:38.981+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:40.207 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:39.919+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:40.207 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:39.919+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:40.207 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:39.919+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:40.207 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:39.919+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:40.208 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:39.966+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:40.208 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:39.966+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:40.208 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:39.966+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:40.208 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:39.966+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:40.208 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:39.997+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:40.208 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:39.997+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:40.209 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:39.997+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:40.209 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:39.998+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:41.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:40.915+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:41.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:40.915+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:41.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:40.915+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:41.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:40.916+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:41.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:40.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:41.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:40.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:41.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:40.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:41.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:40.938+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:41.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:40.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:41.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:40.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:41.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:40.970+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:41.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:40.970+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:42.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:41.956+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:42.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:41.956+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:42.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:41.956+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:42.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:41.957+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:42.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:41.904+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:42.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:41.904+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:42.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:41.904+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:42.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:41.905+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:42.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:41.971+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:42.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:41.971+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:42.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:41.971+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:42.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:41.972+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:43.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:42.974+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:43.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:42.974+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:43.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:42.974+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:43.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:42.974+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:42.936+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:42.936+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:43.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:42.936+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:43.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:42.937+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:43.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:42.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:43.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:42.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:43.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:42.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:43.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:42.984+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:43.804 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:43.524+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 27152 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:44.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:43.996+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:44.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:43.996+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:44.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:43.996+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:44.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:43.997+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:44.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:43.959+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:44.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:43.959+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:44.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:43.959+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:44.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:43.960+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:44.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:43.949+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:44.106 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:43.949+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:44.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:43.949+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:44.107 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:43.949+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:45.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:45.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:45.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:45.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:45.038+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:44.946+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:44.946+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:45.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:44.946+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:45.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:44.947+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:45.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:44.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:45.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:44.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:45.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:44.922+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:45.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:44.923+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:46.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:46.018+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:46.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:46.018+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:46.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:46.018+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:46.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:46.019+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:45.992+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:46.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:45.992+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:46.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:45.992+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:46.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:45.993+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:46.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:45.898+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:46.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:45.898+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:46.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:45.898+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:46.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:45.899+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:47.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:47.066+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:47.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:47.066+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:47.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:47.066+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:47.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:47.067+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:47.022+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:47.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:47.022+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:47.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:47.022+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:47.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:47.023+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:46.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:46.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:47.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:46.897+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:47.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:46.898+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:48.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:47.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:48.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:47.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:48.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:47.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:48.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:47.881+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:48.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:48.082+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:48.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:48.082+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:48.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:48.082+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:48.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:48.082+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:48.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:48.031+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:48.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:48.031+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:48.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:48.031+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:48.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:48.032+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:48.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:48.526+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 27243 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:49.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:49.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:49.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:49.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:49.058+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:48.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:49.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:48.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:49.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:48.878+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:49.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:48.878+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:49.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:49.063+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:49.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:49.063+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:49.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:49.063+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:49.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:49.064+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:50.207 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:50.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:50.207 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:50.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:50.207 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:50.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:50.207 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:50.097+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:50.208 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:50.063+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:50.208 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:50.063+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:50.208 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:50.063+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:50.208 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:50.064+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:50.208 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:49.921+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:50.208 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:49.921+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:50.209 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:49.921+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:50.209 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:49.921+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:51.077+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:51.077+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:51.077+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:51.078+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:51.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:50.873+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:51.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:50.873+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:51.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:50.873+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:51.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:50.874+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:51.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:51.048+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:51.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:51.048+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:51.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:51.048+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:51.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:51.049+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:52.061 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:51.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:52.061 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:51.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:52.062 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:51.880+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:52.062 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:51.881+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:52.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:52.093+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:52.093+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:52.093+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:52.094+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:52.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:52.060+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:52.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:52.060+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:52.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:52.060+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:52.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:52.061+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:53.063 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:52.906+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:53.063 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:52.906+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:53.063 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:52.906+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:53.063 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:52.906+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:53.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:53.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:53.097+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:53.097+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:53.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:53.098+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:53.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:53.062+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:53.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:53.062+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:53.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:53.062+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:53.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:53.062+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:53.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:53.530+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 27334 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:54.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:53.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:53.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:53.940+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:54.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:53.941+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:54.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:54.090+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:54.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:54.090+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:54.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:54.090+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:54.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:54.091+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:54.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:54.068+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:54.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:54.068+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:54.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:54.068+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:54.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:54.068+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:54.903+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:54.903+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:54.903+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:55.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:54.904+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:55.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:55.061+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:55.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:55.062+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:55.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:55.062+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:55.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:55.063+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:55.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:55.056+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:55.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:55.056+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:55.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:55.056+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:55.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:55.057+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:56.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:56.069+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:56.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:56.069+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:56.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:56.069+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:56.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:56.070+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:56.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:56.099+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:56.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:56.099+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:56.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:56.099+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:56.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:56.100+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:56.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:55.885+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:56.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:55.885+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:56.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:55.885+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:56.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:55.886+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:56.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:56.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:56.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:57.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:56.934+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:57.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:57.074+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:57.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:57.074+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:57.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:57.074+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:57.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:57.075+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:57.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:57.066+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:57.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:57.066+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:57.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:57.066+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:57.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:57.066+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:58.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:58.035+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:58.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:58.035+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:58.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:58.035+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:58.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:58.035+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:58.093+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:58.093+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:58.093+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:58.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:58.094+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:58.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:57.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:58.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:57.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:58.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:57.973+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:58.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:57.974+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:58.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:45:58.533+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 27424 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:45:59.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:59.080+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:59.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:59.080+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:59.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:59.080+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:45:59.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:45:59.081+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:59.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:59.068+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:59.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:59.068+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:59.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:59.068+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:45:59.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:45:59.069+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:45:59.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:58.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:59.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:58.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:59.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:58.934+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:45:59.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:58.935+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:00.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:00.061+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:00.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:00.061+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:00.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:00.061+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:00.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:00.062+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:00.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:00.050+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:00.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:00.050+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:00.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:00.050+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:00.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:00.051+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:00.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:59.974+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:00.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:59.974+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:00.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:59.974+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:00.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:45:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:45:59.975+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:01.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:01.071+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:01.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:01.071+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:01.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:01.071+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:01.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:01.072+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:01.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:01.070+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:01.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:01.070+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:01.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:01.070+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:01.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:01.070+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:01.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:00.942+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:01.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:00.942+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:01.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:00.942+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:01.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:00.943+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:02.044 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:01.948+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:02.044 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:01.948+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:02.044 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:01.948+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:02.044 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:01.949+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:02.052+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:02.052+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:02.052+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:02.052+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:02.043+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:02.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:02.043+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:02.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:02.043+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:02.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:02.044+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:02.927+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:02.927+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:02.927+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:03.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:02.928+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:03.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:03.045+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:03.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:03.045+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:03.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:03.045+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:03.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:03.045+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:03.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:03.008+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:03.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:03.009+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:03.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:03.009+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:03.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:03.009+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:03.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:03.536+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 27514 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:04.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:04.034+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:04.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:04.034+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:04.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:04.034+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:04.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:04.034+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:04.023+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:04.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:04.023+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:04.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:04.023+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:04.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:04.023+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:04.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:03.885+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:04.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:03.885+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:04.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:03.885+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:04.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:03.886+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:05.308 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:05.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:05.308 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:05.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:05.308 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:05.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:05.309 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:05.068+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:05.309 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:05.028+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:05.309 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:05.028+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:05.309 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:05.028+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:05.309 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:05.029+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:05.310 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:04.910+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:05.310 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:04.910+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:05.310 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:04.910+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:05.310 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:04.911+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:05.946+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:05.946+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:05.946+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:06.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:05.947+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:06.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:06.029+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:06.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:06.029+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:06.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:06.029+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:06.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:06.029+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:06.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:06.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:06.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:06.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:06.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:06.113+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:06.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:06.113+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:06.936+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:06.936+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:06.936+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:07.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:06.936+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:07.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:07.031+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:07.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:07.031+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:07.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:07.031+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:07.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:07.032+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:07.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:07.127+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:07.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:07.127+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:07.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:07.127+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:07.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:07.127+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:07.935+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:07.935+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:07.935+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:08.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:07.936+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:08.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:08.022+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:08.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:08.022+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:08.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:08.022+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:08.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:08.022+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:08.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:08.080+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:08.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:08.080+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:08.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:08.080+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:08.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:08.080+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:08.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:08.539+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 27604 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:09.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:09.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:09.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:09.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:09.070+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:09.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:09.056+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:09.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:09.056+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:09.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:09.056+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:09.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:09.056+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:09.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:08.931+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:09.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:08.931+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:09.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:08.931+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:09.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:08.931+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:10.208 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:10.046+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:10.208 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:10.046+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:10.208 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:10.046+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:10.208 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:10.047+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:10.209 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:10.073+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:10.209 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:10.073+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:10.209 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:10.073+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:10.209 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:10.074+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:10.209 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:09.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:10.209 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:09.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:10.209 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:09.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:10.210 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:09.888+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:11.012+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:11.012+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:11.012+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:11.013+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:11.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:11.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:11.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:11.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:11.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:11.103+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:11.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:11.104+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:11.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:10.874+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:11.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:10.874+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:11.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:10.874+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:11.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:10.875+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:12.042 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:11.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:12.043 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:11.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:12.043 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:11.893+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:12.043 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:11.893+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:12.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:12.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:12.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:12.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:12.070+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:12.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:12.041+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:12.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:12.042+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:12.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:12.042+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:12.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:12.042+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:13.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:12.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:13.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:12.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:13.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:12.920+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:13.093 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:12.921+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:13.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:13.091+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:13.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:13.091+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:13.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:13.091+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:13.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:13.092+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:13.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:13.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:13.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:13.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:13.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:13.096+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:13.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:13.097+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:13.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:13.542+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 27693 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:14.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:14.047+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:14.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:14.047+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:14.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:14.047+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:14.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:14.048+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:14.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:14.062+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:14.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:14.062+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:14.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:14.062+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:14.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:14.063+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:14.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:13.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:14.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:13.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:14.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:13.887+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:14.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:13.887+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:15.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:15.087+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:15.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:15.087+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:15.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:15.087+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:15.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:15.088+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:15.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:15.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:15.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:15.107+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:15.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:14.888+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:15.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:14.888+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:15.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:14.888+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:15.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:14.889+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:16.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:16.108+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:16.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:16.108+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:16.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:16.108+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:16.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:16.108+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:16.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:16.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:16.095+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:16.096+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:16.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:15.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:16.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:15.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:16.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:15.865+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:16.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:15.865+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:17.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:17.142+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:17.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:17.142+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:17.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:17.142+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:17.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:17.143+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:17.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:17.131+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:17.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:17.131+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:17.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:17.131+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:17.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:17.132+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:17.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:16.894+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:17.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:16.894+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:17.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:16.894+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:17.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:16.895+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:18.159 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:18.157+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:18.160 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:18.157+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:18.160 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:18.157+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:18.160 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:18.140+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:18.161 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:18.140+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:18.161 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:18.140+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:18.161 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:18.140+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:18.161 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:17.908+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:18.161 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:17.908+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:18.161 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:17.908+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:18.161 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:17.909+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:18.546 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:18.158+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:18.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:18.545+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 27784 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:18.952+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:18.952+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:18.952+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:19.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:18.953+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:19.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:19.123+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:19.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:19.123+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:19.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:19.123+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:19.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:19.123+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:19.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:19.118+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:19.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:19.118+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:19.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:19.118+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:19.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:19.119+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:20.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:20.143+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:20.143+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:20.143+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:20.144+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:20.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:20.127+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:20.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:20.127+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:20.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:20.127+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:20.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:20.128+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:20.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:19.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:20.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:19.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:20.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:19.982+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:20.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:19.983+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:20.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:20.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:20.985+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:21.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:20.986+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:21.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:21.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:21.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:21.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:21.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:21.106+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:21.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:21.107+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:21.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:21.147+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:21.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:21.147+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:21.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:21.147+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:21.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:21.147+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:22.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:21.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:22.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:21.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:22.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:21.983+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:22.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:21.983+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:22.150+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:22.150+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:22.150+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:22.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:22.150+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:22.605 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:22.136+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:22.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:22.136+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:22.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:22.136+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:22.606 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:22.137+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:22.780 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-02-10T06:46:22.785 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:46:22.786 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-02-10T06:46:22.787 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-10T06:46:22.799 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T06:46:22.811 INFO:tasks.cephadm:Teardown begin 2024-02-10T06:46:22.811 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1845, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-10T06:46:22.816 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:46:22.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:46:22.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:46:23.091 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:23.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:23.091 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:23.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:23.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:23.022+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:23.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:23.022+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:23.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:23.022+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:23.092 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:23.023+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:23.090+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:23.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:23.091+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:23.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:23.149+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:23.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:23.149+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:23.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:23.149+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:23.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:23.150+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:23.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:23.548+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 27874 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:24.070+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:24.070+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:24.070+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:24.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:24.071+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:24.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:24.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:24.069+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:24.070+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:24.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:24.196+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:24.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:24.196+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:24.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:24.196+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:24.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:24.197+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:25.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:25.072+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:25.072+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:25.072+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:25.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:25.072+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:25.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:25.151+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:25.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:25.151+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:25.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:25.151+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:25.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:25.152+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:25.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:25.032+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:25.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:25.032+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:25.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:25.032+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:25.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:25.033+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:25.916 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:46:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:26.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:26.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:26.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:26.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:26.102+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:26.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:26.200+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:26.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:26.200+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:26.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:26.200+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:26.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:26.201+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:26.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:26.010+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:26.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:26.010+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:26.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:26.010+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:26.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:26.011+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:27.232 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:26.972+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:27.232 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:26.972+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:27.233 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:26.972+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:27.233 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:26.973+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:27.233 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:27.092+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:27.233 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:27.092+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:27.233 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:27.092+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:27.233 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:27.093+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:27.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:27.231+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:27.231+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:27.231+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:27.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:27.232+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:28.280 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:28.104+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:28.280 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:28.104+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:28.280 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:28.104+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:28.280 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:28.105+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:28.281 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:28.005+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:28.281 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:28.005+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:28.281 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:28.005+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:28.281 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:28.006+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:28.552 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:28.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:28.552 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:28.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:28.553 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:28.279+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:28.553 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:28.280+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:28.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:28.551+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 27965 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:29.292 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:29.095+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:29.292 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:29.095+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:29.292 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:29.095+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:29.292 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:29.095+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:29.293 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:28.967+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:29.293 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:28.967+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:29.293 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:28.967+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:29.293 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:28.968+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:29.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:29.290+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:29.290+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:29.290+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:29.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:29.291+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:30.209 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:30.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:30.209 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:30.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:30.209 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:30.062+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:30.209 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:30.063+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:30.210 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:29.920+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:30.210 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:29.920+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:30.210 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:29.920+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:30.210 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:29.921+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:30.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:30.249+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:30.249+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:30.249+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:30.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:30.250+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:31.211 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:31.087+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:31.211 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:31.087+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:31.211 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:31.087+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:31.211 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:31.088+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:31.211 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:30.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:31.212 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:30.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:31.212 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:30.937+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:31.212 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:30.937+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:31.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:31.210+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:31.210+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:31.210+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:31.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:31.211+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:32.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:31.903+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:31.903+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:31.903+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:32.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:31.903+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:32.209+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:32.209+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:32.209+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:32.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:32.210+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:32.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:32.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:32.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:32.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:32.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:32.130+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:32.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:32.132+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:33.103 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:32.891+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:33.103 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:32.891+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:33.103 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:32.891+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:33.103 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:32.892+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:33.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:33.101+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:33.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:33.102+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:33.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:33.102+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:33.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:33.103+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:33.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:33.226+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:33.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:33.226+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:33.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:33.226+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:33.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:33.227+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:33.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:33.554+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28056 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:34.198 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:33.880+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:34.198 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:33.880+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:34.198 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:33.880+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:34.198 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:33.881+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:34.199 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:34.119+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:34.199 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:34.119+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:34.199 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:34.119+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:34.199 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:34.119+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:34.197+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:34.197+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:34.197+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:34.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:34.197+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:34.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:46:34.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:46:35.119 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:34.864+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:35.120 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:34.864+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:35.120 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:34.864+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:35.120 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:34.865+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:35.238+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:35.238+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:35.238+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:35.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:35.239+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:35.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:35.119+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:35.119+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:35.119+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:35.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:35.120+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:36.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:35.844+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:35.844+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:35.844+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:35.845+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:36.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:36.157+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:36.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:36.157+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:36.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:36.157+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:36.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:36.157+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:36.223+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:36.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:36.223+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:36.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:36.223+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:36.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:36.224+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:37.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:36.817+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:37.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:36.817+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:37.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:36.817+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:37.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:36.818+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:37.187+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:37.187+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:37.187+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:37.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:37.188+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:37.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:37.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:37.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:37.110+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:37.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:37.111+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:37.980 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:46:38.087 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:37.800+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:38.088 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:37.800+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:38.088 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:37.800+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:38.088 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:37.801+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:38.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:38.170+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:38.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:38.170+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:38.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:38.170+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:38.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:38.171+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:38.087+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:38.087+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:38.087+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:38.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:38.087+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:38.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:38.557+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28145 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:38.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:38.789+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:38.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:38.789+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:38.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:38.789+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:38.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:38.790+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:39.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:39.175+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:39.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:39.175+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:39.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:39.175+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:39.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:39.177+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:39.078+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:39.078+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:39.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:39.078+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:39.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:39.079+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:40.034+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:40.034+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:40.034+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:40.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:40.035+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:40.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:39.788+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:40.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:39.788+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:40.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:39.788+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:40.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:39.789+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:40.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:40.203+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:40.203+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:40.203+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:40.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:40.203+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:41.033 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:40.745+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:41.033 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:40.745+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:41.033 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:40.745+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:41.033 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:40.746+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:41.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:41.175+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:41.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:41.175+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:41.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:41.175+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:41.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:41.176+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:41.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:41.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:41.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:41.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:41.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:41.032+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:41.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:41.033+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:42.064 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:41.788+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:42.064 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:41.788+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:42.064 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:41.788+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:42.064 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:41.789+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:42.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:42.063+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:42.063+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:42.063+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:42.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:42.063+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:42.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:42.204+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:42.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:42.204+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:42.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:42.204+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:42.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:42.205+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:43.043 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:42.755+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:43.044 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:42.755+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:43.044 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:42.755+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:43.044 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:42.756+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:43.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:43.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:43.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:43.042+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:43.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:43.043+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:43.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:43.203+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:43.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:43.203+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:43.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:43.203+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:43.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:43.203+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:43.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:43.560+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28234 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:43.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:43.730+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:43.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:43.730+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:43.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:43.730+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:43.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:43.730+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:44.064+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:44.064+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:44.064+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:44.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:44.065+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:44.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:44.242+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:44.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:44.242+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:44.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:44.242+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:44.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:44.243+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:45.079 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:44.726+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:45.080 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:44.726+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:45.080 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:44.726+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:45.080 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:44.727+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:45.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:45.207+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:45.207+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:45.207+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:45.208+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:45.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:45.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:45.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:45.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:45.079+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:45.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:45.079+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:45.682+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:45.682+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:45.682+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:45.683+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:46.222+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:46.222+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:46.222+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:46.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:46.223+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:46.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:46.122+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:46.122+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:46.122+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:46.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:46.123+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:46.712+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:46.712+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:46.712+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:47.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:46.713+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:47.206+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:47.206+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:47.206+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:47.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:47.207+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:47.165+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:47.165+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:47.165+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:47.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:47.166+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:47.700+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:47.700+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:47.700+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:47.701+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:48.564 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:48.223+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:48.565 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:48.223+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:48.565 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:48.223+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:48.565 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:48.224+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:48.565 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:48.181+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:48.565 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:48.181+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:48.565 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:48.181+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:48.566 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:48.182+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:48.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:48.563+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28325 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:48.654+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:48.654+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:48.654+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:48.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:48.655+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:49.178+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:49.178+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:49.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:49.178+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:49.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:49.179+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:49.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:49.190+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:49.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:49.190+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:49.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:49.190+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:49.607 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:49.191+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:49.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:46:49.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:46:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:49.659+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:49.659+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:49.659+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:49.660+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:50.173+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:50.173+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:50.173+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:50.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:50.174+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:50.216+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:50.216+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:50.216+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:50.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:50.216+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:51.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:50.692+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:50.692+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:50.692+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:50.693+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:51.388 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:51.125+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:51.388 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:51.125+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:51.388 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:51.125+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:51.389 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:51.126+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:51.389 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:51.252+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:51.389 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:51.252+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:51.389 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:51.252+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:51.389 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:51.252+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:52.094 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:51.674+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:52.094 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:51.674+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:52.094 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:51.674+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:52.094 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:51.675+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:52.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:52.093+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:52.093+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:52.093+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:52.093+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:52.227+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:52.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:52.227+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:52.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:52.227+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:52.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:52.227+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:53.056 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:46:53.073 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:52.655+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:53.073 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:52.655+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:53.073 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:52.655+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:53.073 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:52.656+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:53.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:53.071+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:53.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:53.071+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:53.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:53.071+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:53.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:53.072+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:53.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:53.246+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:53.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:53.246+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:53.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:53.246+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:53.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:53.247+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:53.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:53.565+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28415 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:53.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:53.688+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:53.688+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:53.688+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:53.856 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:53.688+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:54.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:54.102+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:54.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:54.102+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:54.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:54.102+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:54.357 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:54.103+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:54.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:54.264+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:54.357 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:54.264+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:54.358 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:54.264+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:54.358 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:54.265+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:54.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:54.641+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:54.641+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:54.641+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:54.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:54.642+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:55.289 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:55.096+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:55.289 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:55.096+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:55.290 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:55.096+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:55.290 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:55.097+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:55.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:55.288+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:55.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:55.288+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:55.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:55.288+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:55.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:55.289+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:56.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:46:56.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:46:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:55.622+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:55.622+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:55.622+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:56.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:55.622+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:56.124 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:46:56.124 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:46:56.125 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T06:46:56.125 DEBUG:teuthology.orchestra.run.smithi089:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T06:46:56.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:46:56.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:46:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:56.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:56.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:56.296+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:56.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:56.297+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:56.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:56.133+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:56.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:56.133+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:56.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:56.133+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:56.606 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:56.133+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:57.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:56.671+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:57.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:56.671+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:57.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:56.671+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:57.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:56.672+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:57.160+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:57.160+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:57.160+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:57.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:57.161+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:57.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:57.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:57.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:57.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:57.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:57.329+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:57.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:57.329+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:58.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:57.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:58.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:57.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:58.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:57.721+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:58.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:57.722+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:58.571 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:58.153+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:58.571 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:58.153+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:58.572 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:58.153+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:58.572 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:58.154+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:58.572 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:58.344+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:58.572 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:58.344+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:58.572 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:58.344+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:58.572 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:58.345+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:58.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:46:58.570+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28505 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:46:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:58.759+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:58.759+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:58.759+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:46:58.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:58.759+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:59.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:46:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:59.121+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:59.121+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:59.121+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:46:59.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:46:59.122+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:46:59.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:59.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:59.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:59.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:59.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:59.299+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:46:59.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:46:59.299+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:59.783+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:59.783+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:00.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:59.783+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:00.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:46:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:46:59.784+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:00.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:00.157+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:00.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:00.157+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:00.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:00.157+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:00.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:00.158+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:00.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:00.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:00.346+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:00.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:00.348+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:00.854 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:00.830+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:00.830+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:00.830+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:00.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:00.831+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:01.604 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:01.140+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:01.140+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:01.140+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:01.605 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:01.141+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:01.353+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:01.353+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:01.353+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:01.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:01.353+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:02.092 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:01.790+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:02.093 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:01.790+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:02.093 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:01.790+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:02.093 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:01.791+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:02.091+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:02.091+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:02.091+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:02.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:02.092+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:02.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:02.340+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:02.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:02.340+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:02.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:02.340+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:02.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:02.341+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:03.069 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:02.830+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:03.069 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:02.830+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:03.069 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:02.830+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:03.069 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:02.831+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:03.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:03.068+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:03.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:03.068+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:03.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:03.068+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:03.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:03.069+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:03.304+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:03.304+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:03.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:03.304+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:03.356 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:03.305+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:03.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:03.573+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28595 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:04.322 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:04.056+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:04.322 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:04.056+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:04.322 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:04.056+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:04.322 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:04.057+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:04.323 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:03.858+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:04.323 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:03.858+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:04.323 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:03.858+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:04.323 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:03.859+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:04.321+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:04.321+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:04.321+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:04.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:04.321+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:05.038+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:05.038+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:05.038+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:05.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:05.038+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:05.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:04.882+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:05.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:04.882+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:05.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:04.882+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:05.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:04.883+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:05.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:05.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:05.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:05.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:05.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:05.358+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:05.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:05.358+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:06.205 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:06.064+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:06.205 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:06.064+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:06.206 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:06.064+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:06.206 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:06.065+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:06.206 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:05.922+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:06.206 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:05.922+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:06.206 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:05.922+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:06.207 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:05.923+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:06.469 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:06.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:06.469 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:06.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:06.469 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:06.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:06.469 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:06.399+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:07.070 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:06.944+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:07.070 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:06.944+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:07.070 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:06.944+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:07.071 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:06.945+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:07.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:07.069+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:07.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:07.069+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:07.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:07.069+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:07.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:07.070+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:07.703 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:07.421+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:07.703 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:07.421+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:07.703 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:07.421+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:07.703 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:07.422+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:08.038 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:07.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:08.039 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:07.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:08.039 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:07.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:08.039 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:07.942+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:08.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:47:08.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:47:08.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:08.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:08.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:08.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:08.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:08.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:08.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:08.038+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:08.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:08.575+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28685 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:08.463+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:08.463+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:08.463+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:08.464+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:09.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:09.026+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:09.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:09.026+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:09.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:09.026+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:09.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:09.027+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:09.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:08.901+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:09.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:08.901+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:09.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:08.901+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:09.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:08.903+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:09.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:09.459+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:09.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:09.460+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:09.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:09.460+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:09.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:09.461+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:10.212 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:10.062+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:10.212 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:10.062+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:10.212 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:10.062+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:10.212 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:10.063+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:10.213 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:09.926+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:10.213 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:09.926+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:10.213 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:09.926+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:10.214 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:09.927+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:10.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:10.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:10.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:10.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:10.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:10.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:10.606 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:10.424+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:11.264 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:47:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:11.027+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:11.027+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:11.027+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:11.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:11.028+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:11.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:10.929+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:11.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:10.929+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:11.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:10.929+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:11.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:10.931+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:11.742 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:11.397+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:11.742 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:11.397+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:11.742 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:11.397+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:11.742 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:11.397+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:11.992 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:11.959+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:11.993 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:11.959+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:11.993 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:11.959+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:11.993 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:11.960+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:12.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:11.991+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:12.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:11.991+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:12.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:11.991+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:12.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:11.992+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:12.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:12.435+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:12.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:12.435+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:12.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:12.435+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:12.804 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:12.436+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:13.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:13.010+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:13.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:13.010+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:13.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:13.010+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:13.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:13.010+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:13.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:12.918+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:13.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:12.918+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:13.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:12.918+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:13.107 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:12.919+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:13.708 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:13.579+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28776 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:13.708 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:13.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:13.708 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:13.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:13.708 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:13.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:13.709 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:13.430+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:13.975 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:13.943+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:13.975 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:13.943+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:13.975 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:13.943+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:13.976 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:13.943+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:14.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:13.974+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:14.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:13.974+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:14.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:13.974+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:14.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:13.975+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:14.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:14.443+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:14.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:14.443+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:14.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:14.443+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:14.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:14.444+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:15.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:15.014+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:15.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:15.014+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:15.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:15.014+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:15.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:15.015+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:14.991+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:14.991+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:14.991+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:15.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:14.992+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:15.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:15.463+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:15.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:15.463+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:15.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:15.463+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:15.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:15.464+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:16.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:15.976+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:16.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:15.976+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:16.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:15.976+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:16.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:15.976+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:16.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:15.962+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:16.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:15.962+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:15.962+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:16.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:15.963+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:16.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:16.502+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:16.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:16.502+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:16.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:16.502+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:16.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:16.503+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:17.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:16.964+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:17.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:16.964+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:17.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:16.964+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:17.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:16.965+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:17.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:16.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:17.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:16.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:17.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:16.942+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:17.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:16.943+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:17.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:17.516+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:17.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:17.516+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:17.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:17.516+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:17.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:17.517+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:18.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:17.930+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:18.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:17.930+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:18.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:17.930+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:18.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:17.931+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:18.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:17.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:18.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:17.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:18.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:17.977+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:18.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:17.978+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:18.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:18.582+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28866 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:18.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:18.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:18.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:18.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:18.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:18.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:18.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:18.475+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:19.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:18.961+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:19.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:18.961+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:19.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:18.961+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:19.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:18.961+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:19.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:18.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:19.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:18.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:19.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:18.996+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:19.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:18.997+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:19.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:19.466+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:19.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:19.466+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:19.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:19.466+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:19.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:19.467+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:19.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:19.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:19.998+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:20.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:19.999+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:20.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:19.989+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:20.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:19.989+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:20.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:19.989+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:20.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:19.990+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:20.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:20.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:20.490+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:20.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:20.491+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:21.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:20.967+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:20.967+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:20.967+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:21.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:20.968+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:21.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:20.946+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:21.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:20.946+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:21.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:20.946+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:21.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:20.947+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:21.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:21.492+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:21.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:21.492+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:21.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:21.492+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:21.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:21.493+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:22.001+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:22.001+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:22.001+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:22.002+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:22.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:21.897+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:22.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:21.897+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:22.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:21.897+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:22.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:21.898+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:22.802 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:22.466+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:22.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:22.466+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:22.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:22.466+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:22.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:22.466+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:23.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:22.909+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:22.909+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:22.909+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:23.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:22.910+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:23.012+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:23.012+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:23.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:23.012+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:23.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:23.013+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:23.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:47:23.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:47:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:23.585+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 28956 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:23.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:23.467+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:23.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:23.467+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:23.467+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:23.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:23.468+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:24.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:23.907+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:24.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:23.907+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:24.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:23.907+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:24.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:23.908+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:23.993+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:23.993+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:23.993+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:24.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:23.994+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:24.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:24.470+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:24.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:24.470+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:24.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:24.470+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:24.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:24.471+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:25.180 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:24.921+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:25.180 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:24.921+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:25.180 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:24.921+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:25.180 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:24.922+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:25.181 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:24.995+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:25.181 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:24.995+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:25.181 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:24.995+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:25.181 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:24.996+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:25.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:25.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:25.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:25.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:25.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:25.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:25.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:25.457+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:26.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:47:26.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:25.967+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:26.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:25.967+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:26.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:25.967+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:26.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:25.968+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:26.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:26.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:26.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:26.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:26.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:26.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:26.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:26.036+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:26.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:26.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:26.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:26.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:26.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:26.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:26.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:26.459+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:27.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:27.007+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:27.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:27.007+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:27.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:27.007+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:27.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:27.007+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:27.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:26.990+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:27.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:26.990+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:27.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:26.990+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:27.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:26.991+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:27.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:27.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:27.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:27.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:27.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:27.423+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:27.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:27.423+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:28.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:28.032+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:28.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:28.032+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:28.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:28.032+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:28.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:28.033+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:28.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:27.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:28.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:27.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:28.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:27.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:28.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:27.970+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:28.714 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:28.588+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 29046 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:28.714 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:28.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:28.715 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:28.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:28.715 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:28.434+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:28.715 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:28.435+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:29.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:47:29.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:47:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:28.964+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:28.964+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:28.964+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:28.965+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:29.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:29.039+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:29.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:29.039+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:29.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:29.039+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:29.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:29.040+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:29.408 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:47:29.408 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:47:29.409 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T06:47:29.411 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555113/remote/smithi089/crash 2024-02-10T06:47:29.412 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/crash -- . 2024-02-10T06:47:29.481 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555113/remote/smithi173/crash 2024-02-10T06:47:29.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:47:29.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:47:29.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:29.462+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:29.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:29.462+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:29.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:29.462+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:29.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:29.464+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:30.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:30.035+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:30.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:30.035+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:30.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:30.035+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:30.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:30.036+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:29.982+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:29.982+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:30.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:29.982+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:30.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:29.984+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:30.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:30.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:30.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:30.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:30.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:30.452+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:30.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:30.454+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:31.325 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:31.027+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:31.325 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:31.027+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:31.325 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:31.027+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:31.325 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:31.027+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:31.326 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:31.047+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:31.326 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:31.047+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:31.326 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:31.047+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:31.326 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:31.048+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:31.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:31.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:31.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:31.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:31.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:31.430+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:31.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:31.431+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:32.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:32.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:32.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:32.067+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:32.068+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:32.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:32.029+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:32.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:32.029+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:32.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:32.029+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:32.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:32.030+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:32.544 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:47:32.802 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:32.391+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:32.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:32.391+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:32.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:32.391+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:32.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:32.391+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:33.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:33.049+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:33.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:33.049+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:33.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:33.049+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:33.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:33.050+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:33.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:33.049+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:33.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:33.049+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:33.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:33.049+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:33.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:33.050+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:33.640 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:33.591+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 29137 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:33.641 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:33.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:33.641 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:33.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:33.641 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:33.379+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:33.641 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:33.379+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:34.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:34.073+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:34.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:34.073+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:34.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:34.073+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:34.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:34.074+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:34.036+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:34.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:34.036+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:34.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:34.036+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:34.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:34.037+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:34.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:34.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:34.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:34.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:34.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:34.399+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:34.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:34.399+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:35.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:35.121+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:35.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:35.121+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:35.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:35.121+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:35.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:35.122+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:35.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:35.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:35.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:35.057+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:35.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:35.058+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:35.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:35.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:35.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:35.429+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:35.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:35.429+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:36.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:36.133+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:36.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:36.133+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:36.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:36.133+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:36.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:36.134+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:36.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:36.058+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:36.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:36.058+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:36.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:36.058+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:36.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:36.059+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:36.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:36.426+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:36.426+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:36.426+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:36.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:36.427+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:37.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:37.097+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:37.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:37.097+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:37.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:37.097+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:37.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:37.097+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:37.016+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:37.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:37.016+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:37.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:37.016+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:37.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:37.017+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:37.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:37.422+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:37.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:37.422+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:37.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:37.422+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:37.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:37.423+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:38.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:38.052+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:38.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:38.052+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:38.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:38.052+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:38.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:38.052+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:38.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:38.005+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:38.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:38.005+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:38.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:38.005+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:38.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:38.005+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:38.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:38.593+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 29227 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:38.470+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:38.470+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:38.470+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:38.471+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:39.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:39.061+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:39.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:39.061+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:39.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:39.061+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:39.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:39.062+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:39.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:39.011+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:39.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:39.011+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:39.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:39.011+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:39.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:39.011+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:39.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:39.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:39.475+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:39.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:39.476+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:40.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:40.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:40.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:40.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:40.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:40.037+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:40.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:40.038+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:40.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:40.026+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:40.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:40.026+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:40.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:40.026+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:40.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:40.026+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:40.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:40.483+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:40.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:40.483+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:40.483+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:40.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:40.484+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:41.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:41.051+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:41.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:41.051+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:41.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:41.051+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:41.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:41.051+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:41.034+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:41.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:41.034+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:41.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:41.034+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:41.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:41.035+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:41.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:47:41.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:47:41.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:41.495+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:41.495+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:41.495+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:41.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:41.496+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:42.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:42.016+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:42.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:42.016+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:42.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:42.016+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:42.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:42.017+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:42.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:42.025+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:42.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:42.025+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:42.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:42.025+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:42.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:42.026+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:42.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:42.493+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:42.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:42.493+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:42.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:42.493+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:42.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:42.494+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:43.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:43.010+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:43.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:43.010+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:43.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:43.010+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:43.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:43.011+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:43.026+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:43.026+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:43.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:43.026+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:43.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:43.027+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:43.724 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:43.597+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 29317 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:43.724 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:43.469+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:43.724 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:43.469+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:43.724 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:43.469+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:43.724 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:43.470+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:44.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:43.985+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:44.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:43.985+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:44.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:43.985+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:44.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:43.986+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:44.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:44.001+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:44.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:44.001+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:44.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:44.001+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:44.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:44.002+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:44.608 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:47:44.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:44.493+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:44.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:44.493+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:44.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:44.493+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:44.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:44.493+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:44.957+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:44.957+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:44.957+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:45.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:44.957+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:45.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:44.997+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:45.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:44.997+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:45.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:44.997+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:45.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:44.998+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:45.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:45.445+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:45.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:45.445+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:45.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:45.445+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:45.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:45.446+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:46.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:46.041+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:46.041+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:46.041+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:46.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:46.041+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:46.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:45.964+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:46.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:45.964+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:46.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:45.964+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:46.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:45.965+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:46.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:46.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:46.458+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:46.459+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:47.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:47.005+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:47.005+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:47.005+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:47.006+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:47.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:46.929+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:47.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:46.929+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:47.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:46.929+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:47.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:46.930+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:47.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:47.471+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:47.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:47.471+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:47.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:47.471+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:47.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:47.472+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:48.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:47.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:48.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:47.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:48.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:47.969+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:48.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:47.970+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:48.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:47.934+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:48.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:47.934+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:48.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:47.934+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:48.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:47.935+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:48.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:48.601+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 29407 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:48.507+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:48.507+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:48.507+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:48.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:48.507+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:49.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:48.991+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:48.991+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:48.991+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:49.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:48.991+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:49.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:48.937+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:49.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:48.937+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:49.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:48.937+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:49.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:48.938+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:49.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:49.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:49.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:49.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:49.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:49.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:49.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:49.545+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:50.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:49.975+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:50.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:49.975+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:50.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:49.975+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:50.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:49.976+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:50.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:50.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:50.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:50.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:50.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:50.035+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:50.357 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:50.036+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:50.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:50.532+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:50.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:50.532+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:50.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:50.532+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:50.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:50.533+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:51.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:51.009+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:51.009+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:51.009+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:51.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:51.010+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:51.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:50.946+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:51.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:50.946+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:51.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:50.946+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:51.356 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:50.947+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:51.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:51.496+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:51.496+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:51.496+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:51.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:51.497+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:52.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:51.898+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:51.898+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:51.898+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:52.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:51.899+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:52.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:51.984+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:52.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:51.984+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:52.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:51.984+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:52.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:51.984+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:52.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:52.533+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:52.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:52.533+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:52.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:52.533+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:52.803 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:52.533+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:53.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:53.027+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:53.027+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:53.027+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:53.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:53.028+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:52.881+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:52.881+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:53.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:52.881+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:18.261308+0000 (oldest deadline 2024-02-10T06:37:38.261308+0000) 2024-02-10T06:47:53.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:52.882+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:53.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:53.604+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 29498 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:53.579+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:53.579+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:53.579+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:53.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:53.580+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:54.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:53.907+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:54.042+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:54.042+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:54.042+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:54.043+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:54.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:54.578+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:54.578+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:54.578+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:54.578+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:55.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:54.934+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:55.043+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:55.043+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:55.043+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:55.044+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:55.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:55.582+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:55.582+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:55.582+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:55.582+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:56.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:55.927+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:56.027+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:56.027+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:56.027+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:56.028+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:56.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:47:56.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:47:56.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:56.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:56.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:56.590+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:56.590+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:57.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:56.912+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:57.048+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:57.048+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:57.048+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:21.355625+0000 (oldest deadline 2024-02-10T06:37:41.355625+0000) 2024-02-10T06:47:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:57.048+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:57.555+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:57.555+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:57.555+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:57.556+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:58.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:57.867+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:58.041+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:58.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:47:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:47:58.607+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 29588 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:47:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:58.602+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:58.602+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:58.602+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:47:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:58.602+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:59.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:58.859+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:47:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:47:59.037+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:47:59.676 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:48:00.055 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:47:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:47:59.856+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:00.055 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:59.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:00.056 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:59.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:00.056 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:59.648+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:00.056 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:47:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:47:59.649+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:00.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:00.054+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:01.039 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:00.859+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:01.039 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:00.663+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:01.040 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:00.663+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:01.040 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:00.663+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:01.040 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:00.664+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:01.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:01.038+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:02.005 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:01.859+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:02.006 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:01.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:02.006 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:01.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:02.006 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:01.647+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:02.006 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:01.648+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:02.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:02.004+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:02.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:48:02.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:48:02.748 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:48:02.748 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:48:02.749 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T06:48:02.749 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-10T06:48:02.783 INFO:teuthology.orchestra.run.smithi089.stdout:1707545673.003188 mon.smithi089 (mon.0) 710 : cluster [WRN] Replacing daemon mds.cephfs.smithi089.mgkvmj as rank 0 with standby daemon mds.cephfs.smithi173.tumnkz 2024-02-10T06:48:02.784 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T06:48:02.784 INFO:tasks.cephadm:Compressing logs... 2024-02-10T06:48:02.784 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T06:48:02.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:48:02.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:48:02.857 INFO:teuthology.orchestra.run.smithi089.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T06:48:02.857 INFO:teuthology.orchestra.run.smithi089.stderr:: No such file or directory 2024-02-10T06:48:02.857 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mon.smithi089.log 2024-02-10T06:48:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi089.evydlq.log 2024-02-10T06:48:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5/var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mon.smithi089.log: --verbose -- /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T06:48:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi089.evydlq.log: gzip -5 --verbose -- /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T06:48:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose/var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph.audit.log: -- /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mgr.smithi089.pwmrye.log 2024-02-10T06:48:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T06:48:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mgr.smithi089.pwmrye.log: gzip -5 --verbose -- /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T06:48:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph.log 2024-02-10T06:48:02.858 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph.cephadm.log: 89.9% -- replaced with /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T06:48:02.859 INFO:teuthology.orchestra.run.smithi089.stderr: 81.4% -- replaced with /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T06:48:02.859 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T06:48:02.859 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T06:48:02.859 INFO:teuthology.orchestra.run.smithi089.stderr: 85.7% -- replaced with /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T06:48:02.859 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi089.mgkvmj.log 2024-02-10T06:48:02.860 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.2.log: /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-volume.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T06:48:02.868 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi089.mgkvmj.log: 90.8% -- replaced with /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T06:48:02.907 INFO:teuthology.orchestra.run.smithi089.stderr: 95.0% -- replaced with /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi089.mgkvmj.log.gz 2024-02-10T06:48:02.908 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:02.904+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:02.908 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:02.629+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:02.908 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:02.629+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:02.908 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:02.629+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:37:26.696256+0000 (oldest deadline 2024-02-10T06:37:46.696256+0000) 2024-02-10T06:48:02.908 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:02.630+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:03.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:03.017+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:03.591 INFO:teuthology.orchestra.run.smithi089.stderr:gzip: /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mgr.smithi089.pwmrye.log: file size changed while zipping 2024-02-10T06:48:03.592 INFO:teuthology.orchestra.run.smithi089.stderr: 89.9% -- replaced with /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mgr.smithi089.pwmrye.log.gz 2024-02-10T06:48:03.613 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:03.587+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:gzip: /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mon.smithi089.log: file size changed while zipping 2024-02-10T06:48:03.868 INFO:teuthology.orchestra.run.smithi089.stderr: 92.6% -- replaced with /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mon.smithi089.log.gz 2024-02-10T06:48:03.925 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:48:03.610+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 29678 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:48:04.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:03.924+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:04.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:04.027+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:04.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:04.598+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:05.248 INFO:teuthology.orchestra.run.smithi089.stderr: 95.3% -- replaced with /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi089.evydlq.log.gz 2024-02-10T06:48:05.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:04.906+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:05.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:05.011+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:05.888 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:48:05.889 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:05.609+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:06.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:05.887+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:06.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:06.033+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:06.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:06.600+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:07.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:06.901+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:07.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:07.018+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:07.484 INFO:teuthology.orchestra.run.smithi089.stderr:gzip: /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-02-10T06:48:07.484 INFO:teuthology.orchestra.run.smithi089.stderr: 95.0% -- replaced with /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T06:48:07.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:07.578+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:07.864 INFO:teuthology.orchestra.run.smithi089.stderr:gzip: /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-02-10T06:48:07.865 INFO:teuthology.orchestra.run.smithi089.stderr: 95.0% -- replaced with /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T06:48:07.940 INFO:teuthology.orchestra.run.smithi089.stderr:gzip: /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-02-10T06:48:07.940 INFO:teuthology.orchestra.run.smithi089.stderr: 95.0% -- replaced with /var/log/ceph/84e5b1ba-c7da-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T06:48:07.942 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-02-10T06:48:07.942 INFO:teuthology.orchestra.run.smithi089.stderr:real 0m5.104s 2024-02-10T06:48:07.942 INFO:teuthology.orchestra.run.smithi089.stderr:user 0m18.459s 2024-02-10T06:48:07.942 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m0.420s 2024-02-10T06:48:08.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:07.894+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:08.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:07 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:07.992+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:08.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:48:08.614+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 29768 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:48:08.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:08.564+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:09.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:08.981+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:09.355 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:08 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:08.874+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:09.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:09.582+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:10.217 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:09 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:09.864+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:10.217 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:10.014+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:10.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:10.818+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:10.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:10 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:10.582+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:11.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:11.057+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:11.822 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:11.553+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:12.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:11 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:11.821+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:12.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:12.045+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:12.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:12.591+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:12.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:12 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:12.809+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:13.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:13.033+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:13.991 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:48:13.617+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 29859 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:48:13.991 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:13.808+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:13.992 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:13.604+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:14.332 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:13 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:13.990+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:14.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:48:14.890 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:48:14.963 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:14.615+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:14.964 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:14.774+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:14.964 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:14.774+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:14.964 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:14.774+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:14.964 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:14.775+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:15.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:14 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:14.962+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:15.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:15.597+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:15.740+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:15.740+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:15.740+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:15.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:15.740+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:16.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:15 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:15.960+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:16.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:16.590+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:16.765+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:16.765+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:16.765+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:16.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:16.766+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:17.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:16 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:16.932+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:17.925 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:17.635+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:17.925 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:17.746+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:17.925 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:17.746+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:17.925 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:17.746+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:17.925 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:17.746+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:17.952 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:48:18.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:17 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:17.924+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:18.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:48:18.621+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 29949 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:48:18.956 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:18.648+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:18.956 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:18.698+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:18.957 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:18.698+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:18.957 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:18.698+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:18.957 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:18.698+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:19.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:18 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:18.955+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:19.943 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:19.626+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:19.944 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:19.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:19.944 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:19.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:19.944 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:19.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:19.944 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:19.743+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:20.218 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:19 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:19.941+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:20.727 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:20.581+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:21.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:20.725+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:21.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:20.725+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:21.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:20.725+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:21.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:20.726+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:20.897+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:21.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:20.897+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:21.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:20.897+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:21.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:20 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:20.898+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:21.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:21.558+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:21.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:21.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:21.679+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:21.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:21.679+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:22.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:21.896+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:21.896+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:21.896+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:22.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:21 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:21.896+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:22.802 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:22.544+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:22.723+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:22.723+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:22.723+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:22.803 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:22.723+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:23.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:22.880+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:22.880+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:22.880+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:23.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:22 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:22.881+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:23.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:48:23.623+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 30039 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:48:23.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:23.571+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:23.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:23.709+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:23.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:23.709+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:23.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:23.709+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:23.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:23.709+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:24.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:23.856+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:23.856+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:23.856+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:24.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:23 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:23.857+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:24.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:24.524+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:24.741+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:24.741+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:24.741+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:24.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:24.742+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:25.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:24.855+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:24.855+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:24.855+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:25.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:24 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:24.856+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:25.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:25.574+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:25.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:25.818+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:25.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:25.818+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:25.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:25.818+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:25.855 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:25.818+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:25.771+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:25.771+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:25.771+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:25.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:25 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:25.772+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:26.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:26.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:26.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:26.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:26.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:26.743+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:26.849 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:26.744+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:26.850 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:26.561+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:26.850 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:26.561+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:26.850 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:26.561+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:26.850 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:26.562+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:27.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:26.848+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:26.848+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:26.848+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:27.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:26 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:26.849+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:27.704+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:27.704+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:27.704+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:27.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:27.705+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:27.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:27.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:27.567+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:27.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:27.568+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:28.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:27.864+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:27.864+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:27.864+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:28.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:27 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:27.864+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:28.854 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:48:28.628+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 30129 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:48:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:28.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:28.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:28.677+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:28.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:28.678+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:28.600+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:28.600+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:28.600+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:28.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:28.601+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:29.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:28.895+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:28.895+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:28.895+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:29.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:28 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:28.896+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:29.918 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:29.697+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:29.918 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:29.697+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:29.919 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:29.697+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:29.919 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:29.698+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:29.919 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:29.606+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:29.919 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:29.606+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:29.919 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:29.606+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:29.919 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:29.607+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:29.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:48:29.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:48:30.219 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:29.917+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:30.219 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:29.917+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:30.219 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:29.917+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:30.219 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:29 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:29.918+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:31.104 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:30.654+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:30.654+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:30.654+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:31.105 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:30.656+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:31.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:30.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:31.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:30.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:31.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:30.659+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:31.106 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:30.660+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:31.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:30.880+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:31.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:30.880+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:31.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:30.880+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:31.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:30 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:30.881+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:31.895 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:31.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:31.895 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:31.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:31.895 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:31.619+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:31.895 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:31.620+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:31.895 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:31.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:31.895 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:31.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:31.896 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:31.610+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:31.896 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:31.611+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:32.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:31.894+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:31.894+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:31.894+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:32.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:31 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:31.894+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:32.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:32.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:32.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:32.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:32.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:32.655+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:32.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:32.656+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:32.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:32.569+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:32.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:32.569+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:32.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:32.569+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:32.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:32.570+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:33.020 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:48:33.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:32.882+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:32.882+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:32.882+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:33.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:32 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:32.883+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:33.621 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:33.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:33.621 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:33.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:33.621 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:33.544+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:33.621 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:33.545+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:33.909 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:48:33.631+0000 7f0e013a9700 -1 mon.smithi089@0(probing) e2 get_health_metrics reporting 30220 slow ops, oldest is monmgrreport(gid 14210, 0 checks, 1 progress events) 2024-02-10T06:48:33.910 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:33.620+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:33.910 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:33.620+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:33.910 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:33.620+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:33.910 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:33.620+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:34.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:33.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:33.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:33.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:34.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:33 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:33.909+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:34.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:34.511+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:34.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:34.511+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:34.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:34.511+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:34.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:34.512+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:34.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:34.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:34.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:34.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:34.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:34.600+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:34.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:34.601+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:34.899+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:34.899+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:34.899+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:35.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:34 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:34.900+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:35.836 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:35.622+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:35.836 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:35.622+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:35.836 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:35.622+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:35.836 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:35.622+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:35.837 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:35.497+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:35.837 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:35.497+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:35.837 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:35.497+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:35.837 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:35.498+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:36.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:48:36.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:48:36.092 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:48:36.092 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:48:36.093 INFO:tasks.cephadm:Removing cluster... 2024-02-10T06:48:36.093 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 --force 2024-02-10T06:48:36.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:35.895+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:36.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:35.895+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:35.895+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:36.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:35 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:35.896+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:36.537 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:36 smithi089 systemd[1]: Stopping Ceph mon.smithi089 for 84e5b1ba-c7da-11ee-95b8-87774f69a715... 2024-02-10T06:48:36.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:48:36.628+0000 7f0e043af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi089 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:48:36.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089[29213]: 2024-02-10T06:48:36.628+0000 7f0e043af700 -1 mon.smithi089@0(probing) e2 *** Got Signal Terminated *** 2024-02-10T06:48:36.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:36.660+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:36.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:36.660+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:36.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:36.660+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:36.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:36.661+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:36.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:36.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:36.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:36.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:36.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:36.536+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:36.858 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:36.537+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:37.240 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:36 smithi089 podman[83872]: 2024-02-10 06:48:36.944220901 +0000 UTC m=+0.436805162 container died 0109d6119759c20e843eb29825aa4cf2e87ada86c2823fffbf74f2e06a447a1a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:48:37.256 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:36.931+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:37.256 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:36.931+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:37.256 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:36.931+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:37.257 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:36 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:36.932+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:37.552 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:37 smithi089 podman[83872]: 2024-02-10 06:48:37.338591215 +0000 UTC m=+0.831175468 container cleanup 0109d6119759c20e843eb29825aa4cf2e87ada86c2823fffbf74f2e06a447a1a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-10T06:48:37.553 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:37 smithi089 bash[83872]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089 2024-02-10T06:48:37.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:37 smithi089 podman[83885]: 2024-02-10 06:48:37.615698836 +0000 UTC m=+0.669610408 container remove 0109d6119759c20e843eb29825aa4cf2e87ada86c2823fffbf74f2e06a447a1a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-mon-smithi089, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=quincy-72ce7c1, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-10T06:48:37.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:37 smithi089 systemd[1]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mon.smithi089.service: Deactivated successfully. 2024-02-10T06:48:37.855 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:37 smithi089 systemd[1]: Stopped Ceph mon.smithi089 for 84e5b1ba-c7da-11ee-95b8-87774f69a715. 2024-02-10T06:48:37.856 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Feb 10 06:48:37 smithi089 systemd[1]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@mon.smithi089.service: Consumed 21.158s CPU time. 2024-02-10T06:48:37.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:37.690+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:37.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:37.690+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:37.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:37.690+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:37.857 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:37.691+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:37.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:37.551+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:37.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:37.552+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:37.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:37.552+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:37.857 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:37.552+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:38.261 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:37.970+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:38.261 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:37.970+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:38.261 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:37.970+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:38.261 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:37 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:37.971+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:38.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:38.514+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:38.514+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:38.514+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:38.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:38.515+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:38.695+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:38.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:38.695+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:38.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:38.695+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:38.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:38.696+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:39.341 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:38.990+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:39.341 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:38.990+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:39.341 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:38.990+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:39.341 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:38 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:38.991+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:39.600 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:39.478+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:39.601 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:39.478+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:39.601 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:39.478+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:39.601 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:39.480+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:39.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:39.720+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:39.720+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:39.720+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:39.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:39.721+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:40.220 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:39.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:40.220 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:39.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:40.220 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:39.949+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:40.221 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:39 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:39.950+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:40.484 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:40.481+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:40.484 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:40.481+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:40.484 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:40.481+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:40.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:40.482+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:40.715+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:40.715+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:40.715+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:40.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:40.716+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:41.242 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:40.939+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:41.242 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:40.939+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:41.243 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:40.939+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:41.243 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:40 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:40.940+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:41.583 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:41.469+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:41.584 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:41.469+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:41.584 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:41.469+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:41.584 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:41.470+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:41.721+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:41.721+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:41.721+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:41.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:41.722+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:42.187 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:41.911+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:42.187 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:41.911+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:42.188 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:41.911+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:42.188 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:41 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:41.912+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:42.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:42.501+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:42.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:42.501+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:42.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:42.501+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:42.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:42.502+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:42.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:42.737+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:42.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:42.737+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:42.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:42.737+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:42.856 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:42.738+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:43.168 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:42.954+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:43.168 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:42.954+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:43.168 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:42.954+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:43.168 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:42 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:42.955+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:43.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:43.524+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:43.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:43.524+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:43.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:43.524+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:43.782 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:43.525+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:44.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:43.780+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:44.036 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:43.780+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:44.037 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:43.780+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:44.037 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:43.780+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:44.037 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:43.909+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:44.037 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:43.909+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:44.037 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:43.909+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:44.037 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:43 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:43.910+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:44.789 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:44.504+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:44.789 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:44.504+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:44.789 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:44.504+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:44.790 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:44.505+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:45.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:44.788+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:44.788+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:44.788+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:45.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:44.789+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:45.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:44.890+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:45.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:44.890+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:45.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:44.890+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:45.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:44 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:44.890+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:45.789 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:45.527+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:45.789 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:45.527+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:45.789 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:45.527+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:45.789 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:45.528+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:46.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:45.787+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:45.787+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:45.787+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:45.788+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:46.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:45 smithi089 systemd[1]: Stopping Ceph osd.0 for 84e5b1ba-c7da-11ee-95b8-87774f69a715... 2024-02-10T06:48:46.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:45.865+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:46.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:45.865+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:46.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:45.865+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:46.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:45 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:45.866+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:46.550 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:46.113+0000 7fde399c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:48:46.551 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:46.113+0000 7fde399c7700 -1 osd.0 47 *** Got signal Terminated *** 2024-02-10T06:48:46.551 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:46.113+0000 7fde399c7700 -1 osd.0 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:48:46.854 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:46.742+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:46.742+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:46.742+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:46.855 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:46.742+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:46.549+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:46.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:46.549+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:46.549+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:46.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:46.550+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:47.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:46.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:46.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:46.874+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:47.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:46 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:46.875+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:47.834 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:47.719+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:47.834 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:47.719+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:47.834 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:47.719+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:47.834 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:47.720+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:47.835 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:47.517+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:47.835 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:47.517+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:47.835 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:47.517+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:47.835 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:47.517+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:48.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:47.833+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:47.833+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:47.833+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:48.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:47 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:47.834+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:48.679 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:48.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:48.679 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:48.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:48.680 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:48.474+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:48.680 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:48.475+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:49.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:48.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:48.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:48.678+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:49.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:48.679+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:48.862+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:49.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:48.862+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:49.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:48.862+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:49.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:48 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:48.863+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:49.832 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:49.691+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:49.833 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:49.691+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:49.833 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:49.691+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:49.833 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:49.692+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:49.833 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:49.446+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:49.833 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:49.446+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:49.833 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:49.446+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:49.834 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:49.447+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:50.104 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:49.831+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:49.831+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:49.831+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:50.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:49 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:49.832+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:50.701 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:50.428+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:50.702 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:50.428+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:50.702 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:50.428+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:50.702 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:50.429+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:51.104 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:50.701+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:50.701+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:50.701+0000 7fde377c8700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:47:53.823646+0000 (oldest deadline 2024-02-10T06:48:13.823646+0000) 2024-02-10T06:48:51.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0[47468]: 2024-02-10T06:48:50.702+0000 7fde377c8700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5759 3.1 3:83dd0a1e:::1000000472e.00000000:head [write 0~855 [1@-1] in=855b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:50.871+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:51.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:50.871+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:51.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:50.871+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:51.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:50 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:50.872+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:51.807 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:51 smithi089 podman[84707]: 2024-02-10 06:48:51.523103363 +0000 UTC m=+5.523519653 container died e372a21351dbe17154e69c32cf59f9a1b7d5f632ff8b9e7f0b960cedb307e0e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T06:48:51.807 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:51.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:51.808 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:51.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:51.808 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:51.456+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:51.808 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:51.457+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:51 smithi089 podman[84707]: 2024-02-10 06:48:51.897612922 +0000 UTC m=+5.898029208 container cleanup e372a21351dbe17154e69c32cf59f9a1b7d5f632ff8b9e7f0b960cedb307e0e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-10T06:48:52.105 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:51 smithi089 bash[84707]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0 2024-02-10T06:48:52.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:51.841+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:52.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:51.841+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:52.105 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:51.841+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:52.106 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:51 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:51.842+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:52.449 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:52 smithi089 podman[84728]: 2024-02-10 06:48:52.157723184 +0000 UTC m=+0.634516290 container remove e372a21351dbe17154e69c32cf59f9a1b7d5f632ff8b9e7f0b960cedb307e0e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-72ce7c1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-10T06:48:52.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:52 smithi089 podman[84818]: 2024-02-10 06:48:52.38977206 +0000 UTC m=+0.018479297 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc 2024-02-10T06:48:52.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:52 smithi089 podman[84818]: 2024-02-10 06:48:52.508948041 +0000 UTC m=+0.137655270 container create 4d6772f6f8a3bfb08f709d76a17d104f944efcf847c6504bbe8ae6fc003af36c (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=quincy-72ce7c1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-10T06:48:52.745 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:52 smithi089 podman[84818]: 2024-02-10 06:48:52.745036293 +0000 UTC m=+0.373743521 container init 4d6772f6f8a3bfb08f709d76a17d104f944efcf847c6504bbe8ae6fc003af36c (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-72ce7c1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-10T06:48:52.746 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:52.448+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:52.746 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:52.448+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:52.746 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:52.448+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:52.746 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:52.449+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:53.049 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:52 smithi089 podman[84818]: 2024-02-10 06:48:52.748796649 +0000 UTC m=+0.377503873 container start 4d6772f6f8a3bfb08f709d76a17d104f944efcf847c6504bbe8ae6fc003af36c (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-10T06:48:53.049 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:52 smithi089 podman[84818]: 2024-02-10 06:48:52.845065537 +0000 UTC m=+0.473772772 container attach 4d6772f6f8a3bfb08f709d76a17d104f944efcf847c6504bbe8ae6fc003af36c (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, RELEASE=quincy-72ce7c1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True) 2024-02-10T06:48:53.050 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:52.886+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:53.050 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:52.887+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:53.050 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:52.887+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:53.050 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:52 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:52.887+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:53.354 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:53 smithi089 podman[84818]: 2024-02-10 06:48:53.049505714 +0000 UTC m=+0.678212947 container died 4d6772f6f8a3bfb08f709d76a17d104f944efcf847c6504bbe8ae6fc003af36c (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-72ce7c1, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:48:53.775 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:53.453+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:53.775 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:53.453+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:53.775 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:53.453+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:53.775 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:53.454+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:54.043 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:53.911+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:54.043 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:53.911+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:54.043 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:53.911+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:54.044 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:53 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:53.912+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:54.044 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:53 smithi089 systemd[1]: Stopping Ceph osd.1 for 84e5b1ba-c7da-11ee-95b8-87774f69a715... 2024-02-10T06:48:54.044 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:53 smithi089 podman[84845]: 2024-02-10 06:48:53.774734028 +0000 UTC m=+0.718471385 container remove 4d6772f6f8a3bfb08f709d76a17d104f944efcf847c6504bbe8ae6fc003af36c (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-72ce7c1, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-10T06:48:54.044 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:53 smithi089 systemd[1]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-10T06:48:54.044 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:53 smithi089 systemd[1]: Stopped Ceph osd.0 for 84e5b1ba-c7da-11ee-95b8-87774f69a715. 2024-02-10T06:48:54.044 INFO:journalctl@ceph.osd.0.smithi089.stdout:Feb 10 06:48:53 smithi089 systemd[1]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.0.service: Consumed 15min 44.319s CPU time. 2024-02-10T06:48:54.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:54.252+0000 7fd7c85f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:48:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:54.252+0000 7fd7c85f3700 -1 osd.1 47 *** Got signal Terminated *** 2024-02-10T06:48:54.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:54.252+0000 7fd7c85f3700 -1 osd.1 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:48:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:54.405+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:54.405+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:54.405+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:54.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:54.406+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:55.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:54.863+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:54.863+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:54.864+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:55.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:54 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:54.864+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:55.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:55.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:55.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:55.356+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:55.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:55.357+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:55.913+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:55.913+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:56.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:55.913+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:56.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:55 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:55.914+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:56.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:56.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:56.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:56.380+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:56.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:56.381+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:57.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:56.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:56.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:56.908+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:57.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:56 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:56.908+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:57.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:57.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:57.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:57.375+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:57.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:57.376+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:58.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:57.870+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:57.870+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:57.870+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:58.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:57 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:57.871+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:58.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:58.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:58.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:58.355+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:58.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:58.356+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:59.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:58.918+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:58.918+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:58.918+0000 7fd7c5bf3700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:00.516872+0000 (oldest deadline 2024-02-10T06:48:20.516872+0000) 2024-02-10T06:48:59.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:58 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1[54771]: 2024-02-10T06:48:58.919+0000 7fd7c5bf3700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5758 3.2 3:46dd91ad:::1000000472d.00000000:head [write 0~718 [1@-1] in=718b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:48:59.758 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:59.395+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:59.759 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:59.395+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:59.759 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:59.395+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:48:59.759 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:48:59 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:48:59.396+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:00.029 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:48:59 smithi089 podman[84935]: 2024-02-10 06:48:59.758533832 +0000 UTC m=+5.629574317 container died 42f862e2f6ca376724d4e7295f1a691d71497480326a88e023d3cb40b3e9f53e (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-72ce7c1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True) 2024-02-10T06:49:00.354 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:00 smithi089 podman[84935]: 2024-02-10 06:49:00.120144473 +0000 UTC m=+5.991184973 container cleanup 42f862e2f6ca376724d4e7295f1a691d71497480326a88e023d3cb40b3e9f53e (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1, RELEASE=quincy-72ce7c1, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2) 2024-02-10T06:49:00.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:00 smithi089 bash[84935]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1 2024-02-10T06:49:00.736 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:00 smithi089 podman[84954]: 2024-02-10 06:49:00.396619326 +0000 UTC m=+0.633407704 container remove 42f862e2f6ca376724d4e7295f1a691d71497480326a88e023d3cb40b3e9f53e (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-72ce7c1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-10T06:49:00.736 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:00.442+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:00.737 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:00.442+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:00.737 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:00.442+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:00.737 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:00 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:00.443+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:01.028 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:00 smithi089 podman[85046]: 2024-02-10 06:49:00.639139949 +0000 UTC m=+0.017709426 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc 2024-02-10T06:49:01.028 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:00 smithi089 podman[85046]: 2024-02-10 06:49:00.772627992 +0000 UTC m=+0.151197448 container create 2f3f56b1db6f15181d045639f30257da1a79469afce8587579064ca0f548d671 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2) 2024-02-10T06:49:01.028 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:01 smithi089 podman[85046]: 2024-02-10 06:49:01.028194871 +0000 UTC m=+0.406764325 container init 2f3f56b1db6f15181d045639f30257da1a79469afce8587579064ca0f548d671 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1-deactivate, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-10T06:49:01.323 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:01 smithi089 podman[85046]: 2024-02-10 06:49:01.031644254 +0000 UTC m=+0.410213710 container start 2f3f56b1db6f15181d045639f30257da1a79469afce8587579064ca0f548d671 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-10T06:49:01.324 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:01 smithi089 podman[85046]: 2024-02-10 06:49:01.127896971 +0000 UTC m=+0.506466440 container attach 2f3f56b1db6f15181d045639f30257da1a79469afce8587579064ca0f548d671 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-10T06:49:01.604 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:01 smithi089 podman[85046]: 2024-02-10 06:49:01.324038324 +0000 UTC m=+0.702607777 container died 2f3f56b1db6f15181d045639f30257da1a79469afce8587579064ca0f548d671 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-10T06:49:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:01.467+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:01.467+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:01.467+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:01.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:01 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:01.468+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:02.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:02 smithi089 systemd[1]: Stopping Ceph osd.2 for 84e5b1ba-c7da-11ee-95b8-87774f69a715... 2024-02-10T06:49:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:01 smithi089 podman[85071]: 2024-02-10 06:49:01.940434707 +0000 UTC m=+0.607089871 container remove 2f3f56b1db6f15181d045639f30257da1a79469afce8587579064ca0f548d671 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0) 2024-02-10T06:49:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:01 smithi089 systemd[1]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-10T06:49:02.355 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:01 smithi089 systemd[1]: Stopped Ceph osd.1 for 84e5b1ba-c7da-11ee-95b8-87774f69a715. 2024-02-10T06:49:02.356 INFO:journalctl@ceph.osd.1.smithi089.stdout:Feb 10 06:49:01 smithi089 systemd[1]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.1.service: Consumed 15min 48.591s CPU time. 2024-02-10T06:49:02.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:02.417+0000 7f2c45eaf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:49:02.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:02.417+0000 7f2c45eaf700 -1 osd.2 47 *** Got signal Terminated *** 2024-02-10T06:49:02.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:02.417+0000 7f2c45eaf700 -1 osd.2 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:49:02.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:02.450+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:02.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:02.450+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:02.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:02.450+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:02.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:02 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:02.451+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:03.756 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:03.462+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:03.756 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:03.462+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:03.756 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:03.462+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:03.757 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:03 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:03.463+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:04.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:04.495+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:04.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:04.495+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:04.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:04.495+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:04.856 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:04 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:04.495+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:05.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:05.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:05.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:05.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:05.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:05.528+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:05.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:05 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:05.528+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:06.854 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:06.498+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6804 osd.3 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:06.498+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:06.498+0000 7f2c43cb0700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.173:6820 osd.5 ever on either front or back, first ping sent 2024-02-10T06:48:06.555682+0000 (oldest deadline 2024-02-10T06:48:26.555682+0000) 2024-02-10T06:49:06.855 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:06 smithi089 ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2[62109]: 2024-02-10T06:49:06.499+0000 7f2c43cb0700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:5760 3.8 3:154c21eb:::1000000472f.00000000:head [write 0~1131 [1@-1] in=1131b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-02-10T06:49:08.343 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:08 smithi089 podman[85151]: 2024-02-10 06:49:08.027238692 +0000 UTC m=+5.724348014 container died 69b16e6f89fbff0e0a948c08d1591bb369da6aa3aa1e3f202691de488fad39ce (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=quincy-72ce7c1) 2024-02-10T06:49:08.344 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:08 smithi089 podman[85151]: 2024-02-10 06:49:08.34368136 +0000 UTC m=+6.040790680 container cleanup 69b16e6f89fbff0e0a948c08d1591bb369da6aa3aa1e3f202691de488fad39ce (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1) 2024-02-10T06:49:08.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:08 smithi089 bash[85151]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2 2024-02-10T06:49:08.605 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:08 smithi089 podman[85180]: 2024-02-10 06:49:08.545147639 +0000 UTC m=+0.521333927 container remove 69b16e6f89fbff0e0a948c08d1591bb369da6aa3aa1e3f202691de488fad39ce (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2) 2024-02-10T06:49:09.249 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:08 smithi089 podman[85275]: 2024-02-10 06:49:08.851159285 +0000 UTC m=+0.017009420 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc 2024-02-10T06:49:09.249 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:08 smithi089 podman[85275]: 2024-02-10 06:49:08.954364791 +0000 UTC m=+0.120214916 container create 1abd00ea8da59bd5803c2c8dcd72a5796659ad715d24ae91ea7fca3269bc3bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:49:09.249 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:09 smithi089 podman[85275]: 2024-02-10 06:49:09.173617418 +0000 UTC m=+0.339467539 container init 1abd00ea8da59bd5803c2c8dcd72a5796659ad715d24ae91ea7fca3269bc3bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=quincy-72ce7c1, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-10T06:49:09.249 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:09 smithi089 podman[85275]: 2024-02-10 06:49:09.176892616 +0000 UTC m=+0.342742740 container start 1abd00ea8da59bd5803c2c8dcd72a5796659ad715d24ae91ea7fca3269bc3bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-10T06:49:09.249 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:09 smithi089 podman[85275]: 2024-02-10 06:49:09.248582373 +0000 UTC m=+0.414432506 container attach 1abd00ea8da59bd5803c2c8dcd72a5796659ad715d24ae91ea7fca3269bc3bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:49:09.604 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:09 smithi089 podman[85275]: 2024-02-10 06:49:09.460239239 +0000 UTC m=+0.626089356 container died 1abd00ea8da59bd5803c2c8dcd72a5796659ad715d24ae91ea7fca3269bc3bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=quincy-72ce7c1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131) 2024-02-10T06:49:09.996 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:09 smithi089 podman[85301]: 2024-02-10 06:49:09.97757078 +0000 UTC m=+0.512595723 container remove 1abd00ea8da59bd5803c2c8dcd72a5796659ad715d24ae91ea7fca3269bc3bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715-osd-2-deactivate, RELEASE=quincy-72ce7c1, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-10T06:49:10.354 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:09 smithi089 systemd[1]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-10T06:49:10.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:09 smithi089 systemd[1]: Stopped Ceph osd.2 for 84e5b1ba-c7da-11ee-95b8-87774f69a715. 2024-02-10T06:49:10.355 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 10 06:49:09 smithi089 systemd[1]: ceph-84e5b1ba-c7da-11ee-95b8-87774f69a715@osd.2.service: Consumed 15min 49.994s CPU time. 2024-02-10T06:49:15.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:49:15.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:49:18.876 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:49:27.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:49:27.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:49:30.944 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:49:42.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:49:42.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:49:46.016 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:49:49.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:49:49.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:49:49.084 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:49:49.084 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:49:49.084 INFO:tasks.cephadm:Teardown complete 2024-02-10T06:49:49.085 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1591, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1845, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1848, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1724, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 180, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 348, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi173 2024-02-10T06:49:49.112 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-10T06:49:49.128 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-10T06:49:49.157 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-10T06:49:49.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T06:49:49.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:49:49.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:49:52.256 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:50:01.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:50:01.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:50:04.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:50:16.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:50:16.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:50:19.388 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:50:22.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:50:22.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:50:22.464 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:50:22.464 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:50:22.513 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi173 2024-02-10T06:50:22.533 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T06:50:22.546 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T06:50:22.546 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:50:22.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:50:22.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:50:22.569 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:50:22.573 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:50:22.573 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-02-10T06:50:22.574 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 +867us[ +867us] +/- 92ms 2024-02-10T06:50:22.574 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -1107us[-1074us] +/- 60ms 2024-02-10T06:50:22.574 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +808us[ +808us] +/- 92ms 2024-02-10T06:50:22.574 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:50:25.632 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:50:34.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:50:34.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:50:37.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:50:49.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:50:49.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:50:52.764 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:50:55.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:50:55.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:50:55.836 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:50:55.836 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:50:55.837 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi173 2024-02-10T06:50:55.853 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T06:50:55.866 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T06:50:55.866 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T06:50:55.877 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T06:50:55.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:50:55.965 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:50:59.040 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:51:08.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:51:08.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:51:11.100 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:51:23.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:51:23.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:51:26.172 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:51:29.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:51:29.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:51:29.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:51:29.244 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:51:29.245 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi173 2024-02-10T06:51:29.254 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T06:51:29.265 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T06:51:29.279 INFO:teuthology.task.internal:Duration was 3089.878590 seconds 2024-02-10T06:51:29.279 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T06:51:29.290 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T06:51:29.291 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T06:51:29.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:51:29.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:51:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T06:51:32.352 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:51:41.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:51:41.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:51:44.412 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:51:56.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:51:56.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:51:59.484 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:52:02.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:52:02.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:52:02.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:52:02.556 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:52:02.557 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi173 2024-02-10T06:52:02.574 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T06:52:02.587 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T06:52:02.588 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T06:52:02.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:52:02.626 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:52:05.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:52:14.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:52:14.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:52:17.760 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:52:29.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:52:29.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:52:32.828 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:52:35.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:52:35.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:52:35.903 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:52:35.903 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:52:35.904 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi173 2024-02-10T06:52:35.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T06:52:35.933 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:52:35.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:52:35.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:52:35.968 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-02-10T06:52:39.004 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:52:48.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:52:48.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:52:51.072 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:53:03.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:53:03.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:53:06.140 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:53:09.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:53:09.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:53:09.212 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:53:09.212 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:53:09.213 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi173 2024-02-10T06:53:09.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T06:53:09.240 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T06:53:09.241 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555113/remote/smithi089 2024-02-10T06:53:09.241 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T06:53:09.284 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555113/remote/smithi173 2024-02-10T06:53:09.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:53:09.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:53:12.348 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:53:21.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:53:21.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:53:24.412 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:53:36.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:53:36.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:53:39.484 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:53:42.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:53:42.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:53:42.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:53:42.556 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:53:42.557 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi173 2024-02-10T06:53:42.574 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T06:53:42.587 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T06:53:42.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T06:53:42.599 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T06:53:42.599 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T06:53:42.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:53:42.606 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:53:42.621 INFO:teuthology.orchestra.run.smithi089.stdout: 265293 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 10 06:49 /home/ubuntu/cephtest 2024-02-10T06:53:42.621 INFO:teuthology.orchestra.run.smithi089.stdout: 265294 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 10 06:52 /home/ubuntu/cephtest/archive 2024-02-10T06:53:42.621 INFO:teuthology.orchestra.run.smithi089.stdout: 265298 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 05:59 /home/ubuntu/cephtest/archive/syslog 2024-02-10T06:53:42.621 INFO:teuthology.orchestra.run.smithi089.stdout: 265300 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 10 05:59 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T06:53:42.621 INFO:teuthology.orchestra.run.smithi089.stdout: 265299 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 10 05:59 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T06:53:42.622 INFO:teuthology.orchestra.run.smithi089.stdout: 544409 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 06:50 /home/ubuntu/cephtest/archive/audit 2024-02-10T06:53:42.622 INFO:teuthology.orchestra.run.smithi089.stdout: 544412 176 -rw------- 1 ubuntu ubuntu 177898 Feb 10 06:50 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-10T06:53:42.622 INFO:teuthology.orchestra.run.smithi089.stdout: 265373 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Feb 10 06:06 /home/ubuntu/cephtest/cephadm 2024-02-10T06:53:42.623 INFO:teuthology.orchestra.run.smithi089.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-10T06:53:45.660 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:53:54.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:53:54.662 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:53:57.728 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:54:09.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:54:09.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:54:12.796 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:54:15.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-10T06:54:15.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T06:54:15.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-10T06:54:15.868 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-10T06:54:15.869 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi173 2024-02-10T06:54:15.885 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T06:54:15.898 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T06:54:15.984 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-02-10T06:54:15.985 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T06:54:16.015 DEBUG:teuthology.nuke:shortname: smithi089 2024-02-10T06:54:16.015 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T06:54:16.039 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555113', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:51:37.384077', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKNWJNri5/M3I44SbKPJiPkkMOmm4H0RVWDm51OdoTrgIXiuOjVCXG9a8WnXhNRl5emMgExYGPXc5QV2oedgbmM='} 2024-02-10T06:54:16.043 DEBUG:teuthology.nuke:shortname: smithi173 2024-02-10T06:54:16.043 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T06:54:16.067 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555113', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:51:37.383183', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0pcWUwoBLylHUgQg13zjimT+TLcLEXzDbZlDAy+TtiwziBPT0LuA34roO+1Pb4MX74UMg/Fkq1knHb4ol8pmg='} 2024-02-10T06:54:16.095 INFO:teuthology.orchestra.console.smithi089:Power off 2024-02-10T06:54:16.096 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T06:54:16.121 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2024-02-10T06:54:16.121 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:54:16.137 INFO:teuthology.orchestra.console.smithi173:Power off 2024-02-10T06:54:16.137 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T06:54:16.147 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-02-10T06:54:16.161 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-02-10T06:54:16.161 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:54:16.185 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-02-10T06:54:20.149 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:54:20.186 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:54:20.259 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-02-10T06:54:20.297 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-02-10T06:54:24.264 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:54:24.297 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:54:24.375 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2024-02-10T06:54:24.408 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-02-10T06:54:24.476 INFO:teuthology.orchestra.console.smithi089:Power off completed 2024-02-10T06:54:24.509 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-02-10T06:54:24.614 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3089.878589630127 failure_reason: 'Command failed on smithi089 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84e5b1ba-c7da-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c ''ceph orch ps''"' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b9da24eefb31450e842527ff80037c76 status: fail success: false 2024-02-10T06:54:24.615 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T06:54:24.698 INFO:teuthology.run:FAIL